DEV Community: Andrew Makarov

A Guide to Developing Augmented Reality Indoor Navigation Applications

Andrew Makarov — Wed, 10 May 2023 07:28:35 +0000

With the demand growing for more complex and higher quality AR software products, many established organizations are shifting toward more specialized solutions to suit their business needs, and AR navigation is one of them.

The guide that follows is not another exciting review of the potential of AR navigation, however. Instead, we will share with you the real possibilities of this technology as well as implementation challenges and ways to overcome them. Please keep in mind that this is not an off-the-shelf solution, but is instead our approach and expertise related to building indoor navigation apps that you can use to create a custom AR navigation system for your business or as the basis for building an AR navigation SDK you can share with your clients.

If this is exactly what you were looking for, let’s get started.

Overview of Indoor Navigation Technologies

Before we can get into augmented reality navigation, let’s talk first about some key indoor navigation technologies. Each technology has benefits and drawbacks depending on whether it is used for indoor or outdoor navigation.

WHY GPS IS NOT ENOUGH FOR INDOOR POSITIONING APP DEVELOPMENT

GPS is great if you need to show something on a large scale. For example, showing a supermarket building in AR. In this case, even if the AR object is located 50 meters away, it will still be fine at this scale. But this is not the case for indoor navigation where such accuracy is not satisfactory. Therefore, GPS is only suitable for outdoor navigation. If we need to determine someone’s position inside a building or in an exact position in a dense downtown area, GPS can only do so much to support AR navigation. For example, it can only define a building and preload data for it.

That leaves three other options for indoor navigation: beacons, visual positioning systems, and visual markers.

BEACONS FOR INDOOR NAVIGATION

Indoor positioning system beacons help devices locate their positions within buildings by measuring signals from these beacons. These provide more accuracy than GPS, but you need to purchase and physically place them in the building, as well as store the location of each beacon to calculate user location. The accuracy can be within 5 meters, which is enough, for example, to navigate the user to the gate at the airport.

VISUAL POSITIONING SYSTEM (VPS)

Visual positioning systems and visual markers are approaches that don’t require expensive new hardware. However, they require additional labor to configure and set up.

Just as we search for landmarks with our eyes, visual positioning systems (VPS) can use your smartphone’s camera to analyze your surroundings and determine your location. One of the most notable applications of VPS is used in Google Maps. By comparing camera data with extensive Street View databases, Google can determine your location and give you on-screen directions. Similar technology is also available for iOS, but for a limited number of cities. You can find the list here.

However, things aren’t so easy indoors. You’ll need to create your own datasets and train artificial intelligence models to recognize those environments. You may face challenges if remodeling is done or if the furniture is moved frequently. But VPS can be used as a supplement to GPS or Beacons.

VISUAL MARKERS FOR INDOOR POSITIONING

A visual marker is an image that the system can recognize. It must be unique so that the computer does not confuse it with something else and asymmetric so that the system understands from which side it is viewing the marker.

Visual markers don’t require expensive AI training, making them a potentially more cost-effective solution. Also, they provide the highest level of accuracy of all indoor navigation options available today. But you should keep in mind that users are required to scan a marker to start navigation, which does not provide as seamless an experience as the GPS we are used to. The further we go from the starting point, the more imprecise the navigation becomes. Therefore, this solution requires periodic rescanning. Marker maintenance is also important, as the markers need to be unobstructed and clean for best recognition.

AR Navigation: How It Works

Augmented reality navigation is an innovative solution that incorporates the above-mentioned technologies for indoor and outdoor navigation. The primary goal of this technology is to provide directions to users onscreen, overlaid on top of real environments seen through the camera of a device like a smartphone or headset. This alleviates the challenge of comparing the real world against a reference like a map, which is more difficult for users to use while navigating.

Thus, there are two parts to AR navigation, the first being the actual navigation and localization, and then the display of AR directions such as text, arrows, and paths on the screen. This second step is actually the easiest part of the whole process, with the more challenging element being identifying the user’s position.

Find insights on how AR navigation works and other promising cases of using AR technology in business in the video below.

Current Limitations of AR Indoor Navigation Technology

AR navigation is a truly exciting technology, but it has limitations that need to be considered. AR indoor navigation can lead us in the right direction. It can tell us which way to get to the right department of a store. It can help us get into the general vicinity of a ward of a hospital, or help get us going in the direction of a particular area of an airport. However, at this stage of indoor positioning technology development, it can’t lead us directly to a certain product on a shelf, or to a particular book at the library. Here we’d need centimeter-accuracy, and that’s just not possible yet. You should take this into account when planning your project.

HIGH PRECISION – HIGH PERFORMANCE

When choosing the technologies you want to use for your AR indoor navigation system, aim for high performance and high precision. Mobile devices need precise positioning data and a lot of processing power to make augmented reality and navigation systems work properly. You’ll need to be willing to make the appropriate investment in the technology needed to make your office, airport, hospital, or other location navigable.

Mobile AR Indoor Navigation: ARCore and ARKit

If getting your customers and teams moving in the right direction with indoor navigation is still a challenge you want to take on given these limitations, you’ll now need to consider the platforms that the client will run on, which are Android (ARCore) and iOS (ARKit) smartphones.

Regardless of how you proceed from here, remember that the mobile platforms available primarily differ in their capability to render and display AR directions. The navigation data and routing information can be handled exactly the same across both platforms.

ARKIT: AR NAVIGATION ON IPHONE AND IPAD

ARKit is backed up by much more reliable hardware and software compared to ARCore. Apple has full control over the production and design of its iPhone and iPad hardware as well as the OS software. Because of this, ARKit’s performance is more optimized. It’s more reliable because there is very little diversity in performance among iPhones and iPads.

ARKIT LOCATION ANCHORS

ARKit 4 introduced location anchors in 2020. This technology utilizes VPS to identify landmarks such as buildings and compares them against Apple Maps Look Around images which are similar to Google Street View pictures. This feature allows developers to affix objects to specific locations rather than in arbitrary locations. This can help users identify landmarks around them with AR navigation and this solution operates with higher accuracy than simply using GPS data on its own. Most interestingly, location matching occurs locally, which is good news for users concerned about privacy.

Another key feature that puts iPhones above Android devices in the AR arena is the LiDAR sensor. This hardware can make AR navigation easier due to its superior depth-sensing capabilities, enabling ARKit to analyze depth in a scene at unprecedented speeds.

Check out our demo to see how ARKit-based AR navigation can be implemented.

ARCORE: AR NAVIGATION ON ANDROID SMARTPHONES

Meanwhile, ARCore devices on Android phones are much more inconsistent because there is a great deal of hardware and software diversity from one Android phone to another. Since there are multiple manufacturers, it is more challenging for Google to come up with a consistent, reliable experience.

Although ARCore may not be as powerful as ARKit due to current hardware limitations and inconsistencies, it is still an incredibly valuable platform. In 2022 there were 133,4 million Android users in the USA, roughly a third of the population. Developing AR navigation solutions for this audience is crucial for remaining competitive.

Our ARCore indoor navigation demo video shows how users can be navigated throughout a building using on-screen AR instructions.

How to Develop an AR Navigation Application

In most cases, AR navigation solutions can be implemented into existing applications. Therefore, what follows will focus more on the navigation model. We’ll discuss a few different approaches and their limitations that will enable you to support your customers and staff. We’ll also discuss developing an SDK to sell to your customers for their own mapping solutions.

CREATE AN INDOOR MAP WITH VISUAL MARKERS

Since we don’t have the luxury of using Google, Apple, or open-source mapping solutions to download or call via APIs, you’ll need to accurately map the environment you want to navigate. Quality and scalability are important factors that can determine the success of your map.

Now that we have a detailed map, it must be populated with metadata to define corridors and rooms. This is called the navigation graph. Only then can we begin analyzing the environment to add visual markers.

It’s important to ensure that the map matches the real world. If you are using visual markers, they need to be placed at specific points marked on the map. If they do not perfectly match, then positioning may be inaccurate.

Overcoming Challenges of Indoor Positioning

The most challenging part of AR indoor navigation development is determining the user’s location. Let’s talk about two approaches to indoor positioning.

APPROACH #1: VISUAL MARKERS

Visual markers are helpful for ensuring IPS (indoor positioning system) accuracy. Each marker has its own unique ID. When a marker is seen, it is compared against the entire list of markers to find a match. The markers used for this system include data about the user’s position in 3D space. This includes what floor the user is on and where they are on that floor, as each floor can be determined by a user’s vertical coordinates.

Once implemented, the user will need to scan a marker to calibrate the application for navigation. How close the user’s camera should be to the barcode depends on the quality of the marker and the surface where it’s placed. In most cases, this should only be a few centimeters, but not too close to make it impossible for the barcode to be recognized.

VISUAL MARKERS: COST EFFECTIVENESS VS CONVENIENCE

The visual marker approach is great for relative cost-effectiveness. However, this does sacrifice some convenience for the end user. Having to scan a barcode to initiate each navigation session can be a little tedious for users. If you’re looking for greater convenience, you should explore a method involving beacons. With beacons, seamless positioning can be used without further action from the user. However, the visual marker approach can, in many cases, provide greater precision than beacons can. When properly calibrated, scanning a visual marker will give your device its exact coordinates.

TOO MANY MARKERS

Another problem to consider is scale. Every marker exists in a database used by the app. The more markers there are, the larger the dataset. The larger the dataset, the more processing time is needed to compare the captured image against that dataset to find a match. If the same app is used across multiple locations with thousands of markers, you have a few options to approach degraded performance.

One potential solution is to use different databases for each location. You can use the same marker style across multiple locations with this strategy. The user can select their location manually or may be able to obtain the location information from the local network.

WHY NOT USE QR CODES INSTEAD OF CUSTOM MARKERS?

QR codes can work for this purpose instead of a custom marker design. However, custom markers have better performance when used with ARKit and ARCore applications. The main reason for this is that custom markers allow the program to take into account the orientation of the camera to the marker. This enables greater accuracy than using data from a QR code alone.

RULES FOR THE BUILDING AR MARKERS

For the marker system to work, the owners of the building will need to approve their placement and appearance. If there are strict design standards in a building, visual markers may be challenging to implement. This is closely related to the following considerations:

The more complexity a visual marker has, the easier it will be for the AR app to recognize it.
Every marker must be unique, otherwise, the program may mistake one marker for another.
Asymmetry will tell the program which side of the marker the user is on.

APPROACH #2: BEACONS AND VISUAL POSITIONING SYSTEMS

We believe that visual markers are one of the best solutions for indoor navigation available now. However, combining approaches together is also another potential solution for various use cases.

For example, let’s imagine that you want to create an indoor navigation system for an airport. You may run into some problems with visual markers. Launching into AR navigation requires the user to scan one of these markers. However, this might cause a problem if a marker is not immediately nearby.

If adding additional markers is not possible, VPS might be a better alternative. By using the environment around instead of markers, the user will be able to initiate AR navigation from anywhere in the building. If various parts of your building, like airport terminals, look very similar, then the system may need more information.

In this case, it may be wise to include beacons. The VPS system may be able to recognize the environment, but a nearby beacon may be able to help the system recognize where it is in the building.

GET PERMISSION

In order to use beacons in a building, you’ll need to obtain permission from the building owner. If you don’t have permission to install and maintain beacons or visual markers, you’ll need to use an alternative technology like VPS that won’t be so accurate.

Drawing Routes for Indoor AR Navigation

Rendering the route is the easy part compared to the challenges of understanding the user’s position in a building. However, there are some important considerations to keep in mind. Using graph theory to create directions with lines or arrows, developers can create visuals to direct users to their destination.

One of the main obstacles to overcome is occlusion. The AR 3D layer content is placed on top of the real world on the user’s screen in most contexts. However, this could be confusing to look at if the lines are drawn on top of or through walls. It’s important to consider occlusion so that users can be guided to their destinations more naturally.

Understanding size and distance is also a vital part of the process. Objects that are farther away appear smaller from the perspective of the user. The most important virtual object for this process is the destination pin. When the user can see the destination pin on their screen, it should appear larger when near and smaller when farther away.

AR Navigation SDK Development

If you’re looking to create an SDK to monetize the finished navigation solution, the project becomes a bit more complicated. In addition to navigation, you will have to think about how to ensure platform compatibility. The bare minimum for compatibility would be Android and iOS. Ideally, you would also be able to support Flutter, React, Native Script, and other platforms.

To do this, much of your attention will need to be spent on writing documentation that will help your customers to conveniently adopt your solution. Other considerations include license keys and payment. In our experience, a project of this scale can take 1.5 to 2 years of development. However, you can start with a proof of concept to make sure that the idea is feasible. In addition, you’ll need experienced developers to work on the project. Creating an AR navigation SDK requires many years of expertise.

Future of Indoor Navigation: Increasing Accuracy and Detail

It’s important to set your expectations correctly when it comes to AR indoor navigation. As it stands right now, modern technology doesn’t have the precision to make indoor navigation work as it does in science fiction. However, that doesn’t have to be the case forever. New technologies and innovations are on the horizon, and the field will continue to advance. We should look forward to what lies ahead, but we must approach indoor navigation realistically if we are to succeed.

ULTRA-WIDEBAND

For example, Ultra-Wideband beacons will help provide more accurate coordinates for devices. However, they use more power than other technologies and their signals cannot penetrate walls. Once their APIs are made available, they may serve well in open indoor environments like airports, where beacons might be mounted on ceilings for better connectivity.

ROOM SCANNING

Room scanning technologies are already making it easier for indoor locations to be mapped in 3D space, meaning that it may only be a matter of time before more accessible visual positioning solutions are possible for businesses.

Mobile App Development Trends and Innovative Features in 2023

Andrew Makarov — Fri, 16 Dec 2022 11:34:04 +0000

Engaging with your audience can be challenging. One of the best ways to do this is to use the same platforms that your potential customers do, and mobile devices are what most of the population interacts with. As of 2021, there were approximately 8.6 billion mobile device subscriptions worldwide, according to Statista.

However, mobile applications aren’t just useful for marketing and ecommerce. They can provide real utility in the world of business and personal productivity. No matter what audience your app is intended for, there’s no denying that innovation is one of the core components of success. A great way to get started is to understand the upcoming mobile app development trends. Let’s talk about them, and as you read, consider how each trend might benefit your business.

Key Mobile App Development Market Insights

According to Grandview Research, the mobile app market worldwide was valued at $187.58 billion in 2021. Estimates predict that this will rise at a CAGR of 13.4% from 2022 to 2030. Grandview Research concludes that one of the driving forces behind the growth of mobile applications is increasing accessibility to the Internet in previously inaccessible areas. Artificial intelligence has made applications more powerful as well, making them a popular choice among consumers.

Gaming, retail and e-commerce, health and fitness, music and social networks remain the most popular areas of use for mobile applications. Entertainment is another major category where mobile apps excel. Netflix, Spotify, YouTube, TikTok, and more dominate this segment, all providing content streaming on demand. These have found more innovative success thanks to advanced ML algorithms that learn from the user’s interests and suggest content that they may enjoy.

Now that we have a deeper understanding of the importance and current state of this space, let’s talk about new trends in mobile app development and how these trends benefit businesses.

1: Provide Immersive Experiences with Augmented Reality

Several years ago, Google experimented with (and at one point attempted to revolutionize) immersive experiences on smartphones with virtual reality using Google Cardboard and Google Daydream. Although this vision was never fully realized due to hardware limitations, the objective of providing immersive experiences for users lives on in the form of more accessible technology such as augmented reality. Augmented reality developers have access to more powerful tools than ever before in 2023, allowing users to do far more than catch Pokémon.

AUGMENTED REALITY FOR MOBILE

AR has a number of benefits when used with mobile devices. Firstly, smartphones are so prolific that they are a great platform for sharing immersive experiences with your audience and are far more practical than bulky, expensive headsets. Apps with AR can yield better user engagement due to more lifelike and interactive elements. Augmented Reality technology also provides practical solutions to problems that can be used beyond consumer spaces, showing potential in business environments as well.

For example, AR can be a powerful solution in navigation. By displaying virtual guides in physical space with the help of a smartphone, users can easily navigate in a building or in an open space to find the required location. Also, AR navigation can be used both indoors and outdoors.

Some of the most important aspects of augmented reality in mobile app development trends include creating 3D objects out of subjects in the real world and utilizing a user’s location to enhance AR experiences. There are challenges that need to be considered, such as the technical difficulty of getting a user’s location. However, artificial intelligence has made it possible to improve location detection. For example, Apple ARKit Location Anchors match camera data to Look Around database images (like Google Street View) to identify the user’s location. This can be more reliable than GPS and allows for more realistic AR experiences.

FURTHER BEYOND: THE METAVERSE

Augmented reality stands to make the metaverse more accessible to the general public. While some companies like Meta are focused on developing and proliferating bulky and expensive virtual reality headsets to advance the metaverse, this may not be the only way forward. Instead, leveraging the power of mobile augmented reality can be more accessible and effective for many metaverse applications.

As an example, a Swiss startup called Hololoot allows users to interact with 3D objects in AR with associated NFTs. Allowing users to combine virtual objects with the real world can help provide more immersive experiences and seamless transitions between reality and the web.

2: Improve Security with Advanced Features

In connection with the growth of cyber attacks and their progression, a crucial trend is to improve mobile security with the help of innovative technologies.

BIOMETRICS

One of the hallmarks of modern smartphones is their advanced biometrics technologies. Nearly all modern smartphones use biometrics like fingerprints, face, and voice recognition. Although a bit rarer, some smartphones have iris scanning and whole-hand scanning.

These features aren’t just useful for securing your device. They can also be used to secure mobile applications, or even authenticate access to external systems. For example, a user could use biometric technology on their phone to authenticate themselves when trying to enter a secure room or safe at a business.

However, it’s worth remembering that biometrics cannot protect you 100% of the time and can also fall victim to attackers. Spoofing methods exist for various biometrics. To reduce the likelihood of successful spoofing, developers can enforce more than one biometric method to be used. For example, a user may be required to verify both a fingerprint and a face scan in order to open a safe or access important data. Also, there are different anti-spoofing techniques that can be used to detect liveness.

OTHER SECURITY CONSIDERATIONS

There are a number of other considerations when creating a mobile app with security in mind. One of the most important principles to use is masking sensitive information where possible. This not only prevents ‘over-the-shoulder’ password theft but also theft by other apps that can see your screen. Mobile developers can also make it so that content of the app is hidden when using the app switcher.

More generally, IPC (inter-process communication) protocols and anti-tampering technologies can prevent hackers from intercepting, modifying, and abusing app features to gain access to data.

3: Increase User Engagement with Beacon Technology

The most notable advantage of mobile applications is that they are, in fact, mobile! Being able to use software anywhere is incredibly useful. However, this advantage becomes even more effective when leveraged for its potential for marketing and engagement. Devices, powered by Bluetooth Low Energy and called beacons, can be used to send data like advertisements and notifications to smartphones. This can help provide unique, location-based experiences for users. For example, users in a store can receive beacon-based notifications with coupons and special offers related to their proximity to certain products.

Another potential use case of beacons and mobile applications is shopping personalization. If we’re going to share offers and suggestions to customers in the store like in the previous example, these suggestions would be best served while attending to that customer’s personal needs and wants. Understanding a consumer’s previous behavior can help determine what suggested offers may be the most effective when a shopper is in a particular part of the store.

This wouldn’t be possible without gathering data, and beacons are well-positioned to achieve this goal. If consumers are browsing the store in range of these beacons, stores can track a shopper’s journey. When looking at shopping on a larger scale, stores can even record heatmaps of which parts of the store get the most traffic.

Other businesses can also leverage this technology for marketing and engagement notifications. For instance, ridesharing businesses can use beacons mounted on vehicles to send helpful reminders to riders in their notifications.

4: Expand Functionality with Mobile Payments

There are a number of different ways for money to exchange hands in the modern world. However, one of the most convenient methods is mobile payments with near-field communication (NFC). This allows for secure transactions that can be processed simply by hovering an unlocked smartphone near an NFC-compatible payment terminal. This has a number of benefits:

Mobile payments are convenient
Reduced contact minimizes the transmission of disease at checkout
NFC payments are secure
It’s fast!

According to Peerbits, NFC payments are estimated to value at $39.8 billion USD by 2026. This is a major motivator for businesses to upgrade to NFC-compatible terminals, as this form of payment is becoming far more popular. Notably, Walmart has experienced harsh criticism for refusing to accept mobile payments aside from their own barcode-based Walmart app payments.

This year, Apple added a Tap to Pay feature to its iOS 16 that allows payment apps to directly accept payments from contactless credit or debit cards, Apple Watch, and smartphones with other digital wallets. This made it possible for every small business to accept electronic payments without the need to install a terminal. All they need is their smartphones.

Mobile payments aren’t just limited to NFC. Consumers can send money to someone else via direct payment applications like Zelle, Cash App, and Venmo. In some cases, these are completely free of fees in the case of Zelle. Supporting these forms of payments often done via mobile devices can make retailers and other businesses more accessible to their customers.

As more and more companies add cryptocurrency as a payment method, mobile crypto payments are gaining popularity. For example, Binance Pay solution enables contactless cross-border payments in cryptocurrency through the marketplace. A business can connect Binance Pay and start accepting payments for their goods or services in crypto.

5: Boost Efficiency with Mobile Apps That Don’t Require Installation

Two new technologies on Android and iOS are innovating how mobile applications work. These are Google Play Instant and Apple App Clips respectively. These features allow users to try limited portions of an application for one of two purposes:

To test out the application before installing the full version.
To only use a specific part of an application that’s needed to save space on the device.

An example of an App Clip for Spin that allows you to quickly rent a scooter and make transactions instantly using Apple Pay without having to open the full app
Source: Apple

It’s also possible to use these smaller, more immediately accessible apps to reduce the barrier to start using an app and increase the overall install rate. For online retailers, real estate, and other ecommerce storefronts, an app download can turn away a potential customer. Allowing users to immediately get into the storefront and begin adding things to their cart allows more conversions to be captured.

Not only can this make apps available faster for users, but it helps them save device space. These innovative mobile features make applications more accessible for users with lower-end devices or devices that are nearly maxed out in storage.

6: Engage Better with Superapps

Increasing the time that active users spend with their services is always a challenge for businesses. In brick-and-mortar retail, shopping malls do everything they can to achieve this. In many malls, retail is only a part of what they offer. Food, beverage, banking, beauty care, and movie theaters are just a few of the things shoppers can access in a mall. In the same vein, businesses are leaning into the ‘superapp’ model to engage with users by offering them a variety of services in the same place.

One of the most notable superapps that use this model is WeChat. Most well-known as a messaging app, WeChat offers far more than communication. Dating, money transfer, financing, and gaming are a few of its features. Other apps like Facebook, Snapchat, and Microsoft Start also offer numerous features beyond their primary purpose.

7 Enhance Scalability with Cloud Computing

Mobile devices are fundamentally less powerful than their desktop and laptop counterparts. Mobile app developers have to face the challenge of designing applications that are optimized for weaker hardware, and this often results in various compromises. This issue is made more complex by the inconsistent hardware specifications of Android devices, which can vary in power from the high end to the low end.

Cloud computing makes it possible for mobile devices to take advantage of powerful hardware from afar. Here are a few notable advantages of mobile cloud computing:

Offloading processing power to the cloud reduces smartphone power consumption
Less mobile device storage is used with cloud computing
Scalability is enhanced greatly

One of the most important advantages in this list is scalability. Updating the backend of the cloud can be done without updating the mobile app, meaning that users will spend less bandwidth downloading important updates. The backend can also be scaled up at any time to meet the needs of the business and its users. The end user may not even notice that a transition took place.

Mobile cloud computing can be used for applications like social media, fintech, and even healthcare. What each of these applications has in common is large amounts of data. For example, social media apps require a great deal of storage and distribution of images and videos. Cloud computing can handle large amounts of data like this far better than mobile devices. After being processed on the cloud, the data can be pushed out in a readable format to a mobile device.

8 Build Smarter Apps with Artificial Intelligence and Machine Learning

Mobile app developers have another useful tool at their disposal: artificial intelligence. According to Mordor Intelligence, AI in mobile markets had a value of $2.14 billion USD in 2021. By 2027, they expect that value will grow to $9.68 billion. These are just a few of the different ways that AI can be leveraged in mobile applications:

Data analysis and learning
Supporting backend roles (image recognition, facial recognition, etc.)
Automatization of tasks (AI virtual assistants)
Conversational features (natural language processing)

Machine learning is best suited for handling large amounts of data. This could be used directly by the user. For example, AI can analyze financial data to give suggestions to users. AI may also play a role in the backend, analyzing user data to learn from their behavior. This enables businesses to automatically suggest products and features that consumers want based on their past behavior.

Another backend role that is taken advantage of by users is AI’s ability to enhance more well-known features. For example, AI powers features like image recognition, facial recognition, and more. These features can be integrated into basic applications like camera apps, which can auto-focus on someone’s face.

9: Create a Larger Ecosystem with the Internet of Things (IoT)

Mobile app development stands to benefit greatly from advancing Internet of Things (IoT) technologies. This technology is focused on networks of computers that work together to achieve goals for businesses and consumers. A mobile application tied into these networks can provide users with valuable information about the system and its components.

For example, manufacturing businesses can use IoT-based mobile apps to provide engineers with real-time information about the status of various machines and predictive maintenance timelines. Meanwhile, retail businesses can take advantage of mobile IoT to track products with beacons, analyze store foot traffic, and forecast times of high demand.

There’s also the potential for smartphones to become the acting network themselves. Instead of getting information from an installed network of computers, a number of mobile devices can share their computing power and sensors for certain purposes like locating a lost Bluetooth device.

10: Mobile App Development: Monolithic vs Microservice Approaches

A recent shift in software development has transitioned from monolithic applications to utilizing microservices. The traditional approach to programming is to write all code and features within the same project and service. When the user downloads the application, all parts of the project are self-contained.

However, a more innovative programming strategy is to split up projects into microservices. Instead of putting all the eggs in one basket, programmers separate the project into pieces. Each piece works with every other piece in tandem. The main advantage of this is that different teams can work on each microservice. Bugs that occur can also be more easily identified and fixed within each piece. This, for example, can make building such projects as multifunctional superapps far easier to work with during development.

Future Innovations in Mobile App Development

As time goes on, mobile technologies will only become smarter and more advanced. Mobile app development is a market that is highly responsive to innovation. Consumers crave new, useful, and accessible technologies. This is why advancing AI, beacons, IoT, NFC, blockchain, and other technologies integrate into mobile app development easily. Because of this, it’s critical for business leaders to follow trends of innovation. Having conversations with experts is also extremely important, as this can lead to solutions that can help lead your product to success.

Augmented Reality in iOS Apps: ARKit Development Guide

Andrew Makarov — Wed, 02 Nov 2022 22:00:00 +0000

Among all augmented reality platforms, ARKit stands above the rest in popularity and performance. Apple’s tight-knit ecosystem, standardized hardware, and high performance of ARKit for augmented reality applications make it highly desirable for AR developers. Choosing to make an app with the native iOS framework unlocks the potential for more advanced AR products.

With extensive experience in building AR-powered apps, we’ve created an ARKit development guide that will help you fill some gaps in the understanding of the specifics of ARKit projects.

ARKit Features for Providing Immersive Experience on iOS

The most important aspect of ARKit is that it is the only framework that unlocks the full potential of iPhone and iPad hardware for augmented reality experiences. Some of the greatest features that ARKit has to offer are object and scene recognition, meshing, geographic tracking, face tracking, and more.

ARKit is often compared to its Android competitor, ARCore. Just like ARKit, ARCore can make it possible to utilize the full potential of an Android device for augmented reality. However, Android devices are far more diverse than Apple, having a variety of different specifications and levels of performance. This makes ARKit’s performance much more predictable and consistent.

If you are only targeting iOS or if you are specifically looking to utilize the full power of Apple devices for AR experiences, ARKit is the way to go. Also, if you are going to create two native apps for iOS and Android, we recommend starting with iOS development because it provides more convenient tools for developers than Android, which means a cost-effective option for checking the feasibility of your idea. Also, it covers more potential users.

How to Create Augmented Reality with ARKit

Augmented reality experiences on iOS devices are accomplished via three steps: tracking, scene understanding, and rendering. AR applications require input from sensors like cameras, accelerometers, and gyroscopes. This data then must be processed to determine the camera’s motion in the real world. Once this is complete, 3D virtual objects can be rendered on top of the real world image and displayed to the user.

More advanced AR applications use depth sensors to understand the layout of the scene. This allows for a better sense of scale and can even support features like occlusion, where objects in the real world exist in front of virtual objects in AR.

However, for most AR applications, the best environment is a well-textured and lit area. A flat surface works best for visual odometry, and a static scene for motion odometry. When the environment does not meet these conditions, ARKit provides users with information on the tracking state. The three possible states are: not available, normal, and limited.

ARKit has plenty of features beyond simply displaying objects on flat surfaces. For instance, ARKit 5 supports vertical surfaces, image recognition, and objects placed in geographic space. In order to see the capabilities of ARKit in action, let’s take a look at several use cases that can help your business carve an innovative path to success and corner the market with ARKit development.

Case Study #1: Outdoor and Indoor AR Navigation

GPS is the king of outdoor navigation. However, it suffers from decreased accuracy when indoors and when obstructed by terrain and tall buildings. However, expanding technologies like Bluetooth beacons, Wi-Fi RTT, and Ultra-Wideband (UWB) are filling the gap. Augmented reality navigation can help users get around using GPS and these other technologies for on-screen directions in the form of virtual elements drawn over the real world.

With AR indoor navigation powered by alternative positioning technologies, navigating shopping malls, convention centers, and airport terminals can become much easier. It can also be used in business contexts in stores and distribution centers to help workers find items and packages that they need.

USING INNOVATIVE TECHNOLOGIES FOR INDOOR POSITIONING

If GPS isn’t available, alternative positioning technologies like Bluetooth, Wi-Fi RTT, and UWB can help the device acquire its precise location. After that, ARKit and the rendering engine take care of the rest.

An often-discussed solution in such cases is an IoT beacon network, but our research has shown that the capabilities of beacons for precise AR navigation are quite limited. In our experiment, the beacon navigation accuracy was 3-5 meters, which would be okay for 2D navigation, but AR requires higher accuracy, preferably up to 1 meter, or even better.

There are various challenges to this approach. The beacon network must be carefully positioned to avoid interference and obstruction. For example, UWB signals cannot pass through walls, people, and plants. Also, it can be expensive and is only useful in certain contexts.

Given that beacons don’t allow us to create accurate indoor AR navigation, let’s look at other options that can help us achieve this goal.

IMAGE RECOGNITION AND MACHINE VISION

Another way to provide precise indoor positioning is to use visual markers as a frame of reference. This works similarly to ARKit Location Anchors, which search for matching images in the Look Around database to find the device’s geographic position. However, instead of searching the Apple Maps database, we create our own database of visual markers. These could be QR codes or any other kind of unique code on any surface. When scanned, these markers communicate location metadata to the device.

With the 3D coordinates obtained from the marker, mobile devices can initiate AR navigation experiences to help users find their way through buildings. This eliminates the need for expensive IoT beacon networks. The only required maintenance would be ensuring that the markers are clean and unobstructed.

RENDERING PATHS WITH AUGMENTED REALITY: THE IMPORTANCE OF OCCLUSION

Drawing content on the screen is easy. Making it blend seamlessly with the real world is much harder. For indoor navigation, a drawn line on the screen is more helpful if it becomes occluded (obstructed) by a wall or door when it curves around a corner. Without occlusion, users can become confused about where the on-screen directions are taking them.

Some solutions include simply drawing an arrow on a screen instead of a full line. However, this isn’t as convenient. Another option is to reduce the render distance. This prevents the line from being drawn too far away and makes it easier to avoid having to account for occlusion.

USING LOCATION ANCHORS FOR OUTDOOR AR NAVIGATION WITH ARKIT

Outdoor AR navigation is generally easier to accomplish than indoor navigation due to more reliable technologies being available, such as GPS. However, image recognition-based VPS (Visual Positioning System) systems can make navigation even more accurate. This makes it much easier to display virtual objects for AR experiences like on-screen directions.

There are some drawbacks to this approach though. For example, rural areas may have street view images but not enough artificial features like tall buildings to use as references. This means that it can be harder to provide on-screen directions. Areas without any images in the database will make this method useless. Because of these limitations, image-based positioning is only useful in more urban areas.

Location Anchors allow developers to ‘pin’ virtual objects in geographic space. Users can then see these virtual objects on their screen after first obtaining their position from image recognition. However, there are some drawbacks to mention for these urban areas. Apple Look Around image data isn’t yet available for all cities. A potential alternative might be a network of beacons or GPS. A beacon network would be far more useful for smaller-scale outdoor settings. For example, beacons could be spread through a factory area to find the right workshop.

Case Study #2: AR Measuring Tools

Thanks to the robust sensors used to provide augmented reality, it’s possible to take accurate measurements of environments without ever pulling out a tape measure. This led to the development of Apple’s RoomPlan framework which uses an iPhone or iPad LiDAR scanner to create 3D floor plans of building interiors. This includes room dimensions and can even recognize types of furniture.

We worked on a similar project creating AR home renovation application for a client in Oregon. This allows users to change wall colors, and add new furniture and other objects. However, in some cases, users may want to use more than one material on the same surface. This requires a bit more work to get around. We chose to develop a custom algorithm with a Graph data structure to make it work as expected. This allows users to carve up any surface into any number of zones and choose a different material for each zone.

Even though in this case we created a custom solution, the release of RoomPlan today can be used for similar projects to make it easier for developers. RoomPlan enables developers to efficiently output room scans in USD or USDZ file formats. These include room dimensions and objects the room contains. These file formats can then be imported into other software for further processing and development. Room scanning works extremely quickly, allowing users to create scans of rooms in seconds.

Case Study #3: Face-Based Augmented Reality

A major advantage of ARKit over ARCore is hardware compatibility and stability. The TrueDepth cameras of the iPhone since the iPhone X have been capable of providing consistent quality in face-based AR. Just like other kinds of AR, this involves tracking, scene understanding, and rendering.

ARKit processes data to result in tracking data, face mesh, and blend shapes. Tracking data gives developers a better sense of how to display content as the subject moves their face. Face mesh is the geometry of the face. Blend shapes are set of the parameters of a detected face in terms of the movements of specific facial features. Each parameter has the corresponding value, which is a floating point number indicating the current position of that feature relative to its neutral configuration, ranging from 0.0 (neutral) to 1.0 (maximum movement).

Face-based AR is used for AR face masks and filters to improve user experience and boost engagement. Virtual try-on solutions are another popular option for consumers and businesses. Face-based AR can help consumers try on products at home like sunglasses and makeup without having to leave the house. Brands of clothing, footwear, accessories and cosmetics have been implementing this top retail trend for a long time, and with the growth of interest in the metaverses, it is becoming even more popular.

NON-AR APPLICATIONS OF FACE-BASED ARKIT TECHNOLOGIES

It’s important to remember that because of this advanced level of fidelity with face tracking, ARKit is not necessarily exclusively useful for AR experiences. It could be used simply for face-tracking applications without augmented reality.

For instance, tracking weight loss is possible through facial measurements. As one gains and loses weight, their face changes width. iPhone True Depth cameras can detect these variations and measure them over time with an app. Also, face-based AR can be useful for controlling on-screen elements through blend shapes and eye-motions. With a little imagination, face-based AR opens up a lot of possibilities with ARKit solutions.

Future of ARKit Development

ARKit has enormous potential thanks to its indoor navigation, facial recognition, room scanning features, and more. The number of active ARKit devices grows every year and reached about 1,368 million in 2022 compared to 1,250 million in 2021, according to Artillery Intelligence. This demonstrates the growing interest of businesses in using ARKit to provide immersive experiences for customers.

Although the technology involved in augmented reality development with ARKit is important, the most critical component of any successful ARKit project is an expert development team. Without clear, innovative objectives, businesses can’t create unique and valuable solutions to problems. Only experienced ARKit developers can turn plans into performant software.

How to Create an Augmented Reality App: Technology Guide 2022

Andrew Makarov — Thu, 22 Sep 2022 13:08:23 +0000

Augmented reality’s use and development have trended up over the past several years. This follows investments in mobile hardware capability, interest in immersive virtual experiences with the metaverse, and rising industry competition. Due to these variables, the augmented reality market is set to reach a value of $502 billion by 2027, growing at a CAGR of 62.7%, according to Research and Markets. If you’re planning to become part of these percentages with your augmented reality project, you should know about all the nuances of AR app development.

In this guide, we’ll discuss the details of augmented reality development in 2022, including the choice of technologies and the development flow.

Types of Augmented Reality Apps

There are a number of different types of AR applications including marker-based, markerless, location-based, and superimposition AR solutions.

MARKER-BASED AR

These applications utilize a particular ‘marker’ like a QR code or other image. 3D content in the app is placed in the world relative to, or on top of the marker. An older, but interesting example of marker-based AR is the PlayStation 3 Wonderbook, a gaming ‘peripheral’ that allows players to view a spell book on their screen. The book rotates and moves when they pick up the device and move it around. The camera uses the patterns on the actual book as a reference to display the AR content, a technology that is often used today on Snapchat and Instagram.

MARKERLESS AR

Instead of using set patterns or codes to trigger the content, markerless AR utilizes a camera to detect environment patterns as well as motion sensors to detect surfaces and place 3D objects. This typically involves a number of different technologies working together, such a

GPS and other location tools
Digital compass
Camera
Accelerometer and Gyroscope
Depth sensors

Latest devices are equipped with depth-sensing hardware (LiDAR, ToF) to improve precision. So markerless AR is powered not only by depth sensors and other positioning data, but also by ML algorithms on top of this data. This allows for a more accurate rendering of 3D content and powers the illusion that digital objects are part of the real world. Apps like Pokemon Go utilize markerless AR.

LOCATION-BASED AR

When users enter a particular location, AR applications can use that data to accurately display virtual content. This is how location-based AR works. Instead of simply displaying an object in relative space, developers can show objects in geographical space for users to observe and interact with.

Technology-wise, location-based AR relies on GPS, a digital compass, and an accelerometer. Moreover, there are several approaches to narrow down the position of the device:

– BLE (Bluetooth low energy) beacons

– VPS (Visual Positioning System)

– Low-range Wi-Fi Direct

– UWB (Ultra-WideBand)

SUPERIMPOSITION AR

This type of AR involves digitally replacing an object or superimposing a virtual object on top of another. For example, an app that can digitally change the color of your couch could be considered superimposition AR. This technique is useful for cause-and-effect demonstrations. For example, a user could point a camera at areas of their city to see what it looked like ten years ago from an archive of Google Maps.

Technologies Used to Develop Augmented Reality

Technologies used in augmented reality app development can depend on a number of factors, like the type of hardware being used, the available power of the device, and what application AR is being used for.

Mobile Augmented Reality Platforms

Smartphones have unique advantages compared to other AR platforms. They’re prevalent in the market and are extremely portable. This makes AR more accessible to many consumers since bulky headsets and elegant smart glasses haven’t quite hit the mainstream just yet. Because of this, mobile AR is a prime target for business applications.

Although mobile AR may not be the most powerful or immersive, it certainly has the potential to be very profitable and is one of the most important augmented reality trends to keep track of. Moreover, mobile AR can be a cost-effective way for business owners to join the metaverse trend. Try-on solutions that allow you to test cosmetics or clothes before buying, AR avatars and filters available to users on a smartphone can help businesses to communicate with customers even in a virtual environment.

There are three approaches toward mobile AR that businesses need to choose from:

Native Android AR applications with ARCore
Native iOS AR applications with ARKit
Cross-platform apps

Native augmented reality app development will allow developers to take advantage of the benefit of the power of a device. In turn, a cross-platform application may not be able to take advantage of powerful native features but minimize the development time. Creating the same app with native code on each platform will be more expensive, but if more power and features are required, it may be more effective. If the application is fairly simple, cross-platform code may be enough.

Despite Android dominating the global OS market, developers on GitHub have historically seemed to prefer ARKit based on the number of repositories over the past several years.

AR DEVELOPMENT FOR IOS DEVICES: ARKIT 5 AND ARKIT 6

LOCATION ANCHORS

Location anchors allow developers to affix virtual objects in the real world by using geographic coordinates. For example, location anchors could display a three-dimensional icon or text in space next to an iconic building. Since location anchors are dependent on Apple Maps data, this means that if the city is not supported by Apple Maps, functionality for location anchors may be limited.

One of the most groundbreaking elements of location anchors is how location is approximated. GPS is simply not precise enough to provide location anchors on a user’s screen. To assist, developers can look around with their camera to allow the device to scan for features on buildings around them. By using these architectural features in conjunction with Apple Maps Look Around data, a user’s location can be better approximated for location anchors.

DEPTH API

Depth API is another valuable feature of ARKit 4 which continues to play an important role in ARKit 5 and ARKit 6. It utilizes one of the most powerful hardware features for AR on a mobile device, the LiDAR scanner on the iPad Pro and Phone 12 Pro, 12 Pro Max, 13 Pro, 13 Pro Max and 14 Pro smartphones. This enables much better analysis of scenes and allows for real-world objects to occlude virtual objects with much better accuracy.

RAYCASTING API

Apple’s Raycasting API enables enhanced object placement in conjunction with depth data. This allows much more accurate placement of virtual objects on a variety of surfaces while taking into account their curvature and angle. For example, it’s possible to use this to place an object on the side of a wall or along the curves of a sofa rather than simply flat on the floor. The LiDAR sensor allows for the scanning of surfaces to take place much more quickly compared to traditional methods.

ARKIT 6 UPDATES

ARKit 6 is about to bring several new features to improve AR experiences on iOS devices. Some of the features that Apple is touting include better motion capture, improvements to camera access, and additional locations for Location Anchors. They also plan to include Plane Anchors, a feature that would allow tracking flat surfaces like tables. This makes it easier for those surfaces to be moved without disrupting the AR experience. ARKit 6 will be launched alongside iOS 16 this fall.

As part of our ARKit research here at MobiDev, we tested using ARKit for gaze tracking. This opens up a number of possibilities for iOS, such as eye-based gestures, vision-based website heatmap analytics, and preventing distracted or drowsy driving.

AR DEVELOPMENT FOR ANDROID DEVICES: ARCORE

In an effort to stay competitive, Google has pushed ARCore far to remain one of the most versatile AR development platforms in the world. Let’s explore some of the features that are used by developers.

CLOUD ANCHORS

This tool allows users to place virtual objects in physical space that can be viewed by other users on their own devices. Google has made sure that Cloud Anchors can be seen by users on iOS devices as well.

On the Android side, ARCore is introducing new features to match ARKit’s advances. ARCore v1.33.0 introduces new Cloud Anchors endpoints and terrain anchors. These both improve the geographic anchoring of virtual objects. Earlier this year, ARCore v.1.31.0 introduced ARCore Geospatial API, which, similar to ARKit Location Anchors, utilizes data from mapping databases. In this case, Google Earth and Street View image data is used to identify the user’s geographic location and display virtual objects in those locations accurately.

The following demo demonstrates how ARCore object detection features for a virtual user manua work.

AR FACES

ARCore enables developers to work with high-quality facial renderings by generating a 468-point 3D model. Masks and filters can be applied after the user’s face is identified. This is one of the most popular use cases for AR app features.

AR IMAGES

Virtual business cards and advertising posters are just a few of the applications that are possible with augmented reality images. 2D markers can be used to implement these features as well as more advanced solutions like AR indoor navigation.

Over the past several years, applications like indoor positioning have gotten even easier since our first experiments, making this technology more feasible for use in the real world.

ARKit vs ARCore Features Comparison

These two frameworks for Android and iOS respectively are nearly identical when viewed from a features-perspective. However, the real difference between these two devices is hardware consistency. Apple iPhone and iPad devices are largely more consistent when it comes to the behavior and capability of their hardware. Meanwhile, Android devices are built by a number of different manufacturers with different specifications. Because of this, it becomes more difficult to deliver a consistent experience across many different Android devices.

Since Android hardware is less consistent, it’s important to keep in mind how powerful your AR experience will be and what devices it should be running on. Should it primarily be running on the latest and greatest Samsung and Pixel devices with high-performance AR features, or should it be less intensive to run on more devices? The choice is yours but experienced AR developers are always there to help you find the best solution.

Cross Platform Mobile AR Development in Unity

If utilizing the full power of native AR frameworks on Android and iOS isn’t necessary and if your goal is faster time to market, cross-platform AR development in Unity may be a good option. Unity AR Foundation is a helpful framework for cross-platform augmented reality app development. Despite not being able to take full advantage of each device, Unity AR Foundation is still fairly powerful. It supports Plane detection, Anchors, Light estimation, 2D image tracking, 3D object tracking, body tracking, Occlusion, and more.

However, there are some missing features depending on the platform you’re on. For example, Unity AR Foundation has limitations with some features for ARCore at the moment, such as 3D object tracking, meshing, 2D & 3D body tracking, collaborative participants, and human segmentation. If you want your app to be cross-platform, you’ll have to keep these missing features in mind.

Web-based Augmented Reality Technologies

On one hand, web AR is an extremely accessible technology because it can run on a multitude of devices without installing any additional software. On the other hand, web AR is very limited in features and power.

Some businesses are already utilizing web AR for technologies like virtual fitting room solutions. For example, Maybelline, L’Oréal and other companies have the option for users to virtually try on cosmetic products using their front-facing camera and web AR software.

Web AR is best implemented for simple tasks like facial recognition filters, changing the appearance or color of an object in a scene like hair, replacing backgrounds for videoconferencing, and more. It’s important to remember these limitations when deciding what platform your application should run on.

Augmented Reality Development For AR Wearables

When we talk about wearable technology for augmented reality, we typically refer to gear like Microsoft HoloLens and more portable and comfortable glasses like Google Glass.

From a software engineering standpoint, Microsoft Hololence’s development is based on the Microsoft technology stack and Azure Cloud.

Check our demo that showcases the capabilities of the Microsoft HoloLens for product presentation and educational purposes.

As for AR Glasses, most of the hardware is Android-based, and manufacturers provide SDKs for engineers to create apps. There are still some substantial aspects to be considered. The first is User Experience and User Interface, as the pattern of using such software is entirely different from what we are used to with smartphones. Then, engineers must be capable of building lightweight and optimized apps, as energy consumption is still the main pain point for many devices.

Where to Start Building AR Applications

The development of any software product begins with the definition of business goals. Only when you clearly understand what results you want to achieve, can a development team help to create a workflow for you that meets your business needs. Traditionally, the process of creating an AR-powered product consists of 5 steps.

The process of building an augmented reality mobile app starts with the pre-contract phase. It’s important for you to discuss the project requirements with developers so that everyone is on the same page about the project’s objectives and scope. It also allows back-and-forth feedback between developers and the business to help them clarify the idea and choose the best possible route to success.

Following this, choosing a technology stack is the next logical step. This is essential for business and technical analysis, as it will decide what platform is the best to use and how the project will be built. Moreover, engineers have to come up with a clear vision on how they plan to achieve targeted results, as bringing AR to a product often has hidden pitfalls to be considered.

After that, prototypes and 3D models are created. Once this is done, more thorough development begins. This involves backend & mobile features, AR modules, and QA. Also, there are a number of different challenges with testing AR features that should be taken into consideration.

Finally, the app is finalized for deployment. The deployment will involve a shift from development to support. The app needs to be updated and supported to be compatible with new SDKs and new device requirements.

MAKING THE MAGIC HAPPEN

Businesses with experience in AR applications have a head start on achieving their goals. However, businesses that don’t have internal augmented reality development teams may find it challenging to achieve their vision without help from experienced augmented reality development professionals. Enlisting the help of experienced AR developers is a great way to build your product and get a return on your investment.

12 Augmented Reality Trends of 2023: New Milestones in Immersive Technology

Andrew Makarov — Thu, 22 Sep 2022 12:50:50 +0000

Innovative technologies transform science fiction into reality, and AR is undoubtedly one of them. Holograms, like in the Star Wars and the Marvel movies, now surround us in the real world, bringing a new immersive experience, and it’s more than just entertainment. Today, augmented reality is an effective business tool.

Across a number of different industries like retail, business, gaming, healthcare, and even the military, augmented reality is used for solving various business challenges. It’s important to keep an eye on these technologies to know where the industry is heading. As we discuss these 12 augmented reality trends making moves in 2023, think about how these solutions may benefit your own business.

Trend #1: Leap into the Metaverse

It’s likely no surprise to you that augmented reality is being used alongside other metaverse technologies. The metaverse has barraged news media over this past year since Facebook’s ‘Meta’ rebranding. However, it’s not just marketing hogwash. One of the goals of metaverse technologies is to strike down the barriers between the digital and physical worlds. Since augmented reality can display virtual objects embedded in our real world, various opportunities emerge for businesses and consumers alike.

AVATARS

If we’re going to bring digital experiences into the real world, AR is a great start. Using body and face tracking, as well as advanced scene depth sensing, companies are already working on camera filters that accomplish this. Geenee AR and Ready Player Me partnered up to make this experience a reality. By inserting your avatar into Geenee’s WebAR Builder software, you can effectively ‘wear’ your avatar on camera. The software also takes into account cosmetic items on your Ready Player Me character, including accessories in the form of NFTs.

This technology isn’t new. It’s been seen in use with apps like Snapchat and Instagram for a long time. However, the innovative element is how the app allows users to drop their avatar that they use on other platforms into the app and use it in AR. In the future, this technology could be used to better hybridize virtual meetings. If one person on your team is using a VR headset to attend a meeting and you’re attending without a VR headset, an AR avatar of the person could represent them at your meeting.

Horizon Workrooms, the “metaverted” meeting rooms, presented by Meta group on VivaTech 2022

Making more cost-effective and powerful AR headsets is the number one barrier to entry here. Time will tell how the technology evolves.

SPATIAL AUDIO

Although it may not seem like an augmented reality technology on the surface, spatial audio is very important for enhancing the immersion of AR experiences. Metaverse technologists are obsessed with including all of our five senses in the process, and our hearing is no exception. To make VR and AR experiences more immersive, 3D audio is needed. Users should be able to tell where a sound is coming from in 3D space based on their own position.

Meta recently added an advanced engine to its AR Spark Studio to create sound effects by mixing multiple sounds. This allows creators to create multi-sensory effects that allow people to use both sight and sound to feel more immersed in the augmented reality experience. In this way, we can make sounds play in response to human interaction with our AR effect.

TAKING DIGITAL ITEMS INTO THE REAL WORLD

Metaverse fans love taking things from the digital world into the real world and vice versa with AR. This technology has actually been around since before the metaverse craze. For example, Meta is working on displaying digital collectibles in AR. All creators will need to do is import their NFTs as 2D virtual objects into Instagram Stories and combine them with “See in AR” functionality. This will open up new opportunities for collectors and creators to access and share their NFTs beyond their wallets and it will certainly quickly become one of the key augmented reality market trends in 2023.

There has also been some buzz about virtual art, or art from the real world being offered as AR experiences. For example, Sotheby’s, the fourth oldest auction house in the world, has begun offering AR experiences to bidders through an Instagram filter that allows them to see art up for auction up close and personal. Sotheby’s used this technology to sell a painting for $121.2 million.

Trend #2: Augmented Reality Meets Artificial Intelligence

There are two ways that artificial intelligence plays nicely with augmented reality:

Artificial intelligence powers facial and spatial recognition software needed for AR to function.
AR and AI solutions can work together to provide innovative solutions.

These roles aren’t necessarily exclusive. They tend to blend together quite a bit.

AI MAKES AR WORK BETTER

Augmented reality and artificial intelligence are separate technologies. However, it’s no surprise that AI and AR work well together due to AR’s needs. Complicated algorithms must be used to make sense of sensor data of the environment. AI can simplify that process and make it more accurate than a model made exclusively by a human.

An example of this in practice is the app ClipDrop. The app allows users to quickly digitize an item in the real world into a 3D object for use in programs like PowerPoint, Photoshop, Google Docs, and more. 3D scanning can be used to import real-world objects into metaverse environments as well. 3D scanning may be a great way for businesses to speed up the pipeline of offering items for virtual trial experiences as well.

Automatic design is another use case of combining AR and AI. An app called SketchAR is an example of this technology in action. Users can freely draw in AR using this app. However, they can also use an AI to draw for them. The AI can create structures quickly. This shows that it’s possible for AI programs to design objects in 3D space using the real world as the source environment. In the future, this may mean that AI will be able to design and create structures for use in the real world.

Trend #3: Mobile Augmented Reality is Evolving

One of the main vehicles for delivering augmented reality experiences has been mobile devices. Most consumers have some kind of mobile device, and AR headsets haven’t gone mainstream for consumer use just yet. Because of that, businesses have found a number of opportunities to leverage mobile devices for AR. The technology has improved significantly as well over the years.

GEOSPATIAL API FOR ARCORE

In 2022, Google introduced a new API for geospatial experiences. This allows developers to create experiences that are tied to specific locations in space. In the past, AR experiences have been purely relative to the user or in arbitrary locations set by the user.

Geospatial API allows developers to set latitude and longitude coordinates for AR content. Scanning the physical space isn’t necessary either. It works very similarly to Apple ARKit Location Anchors, comparing images of the surrounding area to Google Street View images to determine a specific location nearly instantaneously.

ARKIT 6 ENHANCEMENTS

Several new features were introduced by Apple for their ARKit 6 upgrade in 2022 at WWDC. One of them is a 4K video recording while ARKit content is in use. The depth API has also received an upgrade to make scene occlusion and other experiences much more realistic. Apple’s LiDAR scanner allows AR experiences to be prepared so quickly that they call the technology ‘instant AR’.

Apple also improves its motion capture feature. When the camera is focused on another person, motion capture data can be taken from their movements and applied to a 3D model. One of the other latest upgrades is people occlusion, which allows virtual objects to pass in front of and behind people in the scene.

One major example of these ARKit 6 enhancements in use is through RoomPlan, a solution that leverages LiDAR scanning to quickly create floor plans of a house or other structure.

ARCORE VS ARKIT

The competition between Apple and Google in the augmented reality arena has been more or less the same over the past several years. As usual, both technologies are on par with one another in terms of software.

However, hardware is where things get more interesting. Apple’s LiDAR scanner and similar technologies found on higher-end Samsung devices can leverage the highest quality AR experiences available. However, there’s a wide variety of hardware differences between Android devices. Many Android devices simply aren’t powerful enough to handle higher-end AR.

Because of this, businesses need to be strategic about the kinds of AR experiences they want to offer. If they want to offer high-quality experiences to a smaller, wealthier audience with the devices that can handle it, they can let their imaginations run wild. However, if a business is looking for an accessible experience for more devices, they will need to tone things down a bit with a simpler app.

Trend #4: WebAR: Better Accessibility with Compromise

Another important trend in augmented reality is WebAR. Powered by web browsers, WebAR doesn’t require users to download additional software. This is the best-case scenario for accessibility. However, it comes at a cost — WebAR offers the most basic AR experiences and lacks many of the features that native AR can offer on mobile devices.

However, in some cases, WebAR can be very useful for simple experiences. Like adding filters to faces, changing the color of hair or objects, background replacement, and simple 3D objects. Simpler virtual try-on experiences are possible with WebAR. These are used by a number of businesses like L’Oréal and Maybelline for their cosmetic products.

Tom Emrich from 8th Wall, the world’s leading WebAR development platform, notes that WebAR is the key to bridging the gap between the virtual and physical worlds. Although WebAR isn’t very powerful at the moment, the evolution of WebAR may be one of the most important ways to engage with the Internet in the future. 8th Wall is continuing to improve WebAR technologies to fulfill this vision.

Trend #5: Cross-Platform AR Gains Prominence

One major challenge in developing AR is making apps cross-platform. There’s also the unfortunate truth that cross-platform applications will most likely not be quite as good as the full potential of native ones. However, cross-platform apps can be very high quality if the right steps are taken. Cross-platform AR is easier to code and can result in a faster time to market. However, performance and presentation can suffer.

Generally, it’s better to keep an app native if the app is very complex and needs to use the full potential of native features. However, if the app is simpler and doesn’t need extremely high performance, cross-platform will do just fine.

For example, if you are creating an online store where 90% of the functionality is not platform-dependent, does not require maximum performance and has a simple product preview module in AR, then you can opt for cross-platform AR. But if we deal with an application whose functionality requires maximum performance or is platform-dependent, then the native option is better. This applies to projects such as 3D scanning or AR navigation.

Working with an augmented reality development company is a great way to build cross-platform applications with the highest quality possible. This allows you not only to improve the quality of your product but also helps you focus on other aspects of your business.

Trend #6: AR Glasses, Future or Fiction?

It seems like with every year that passes, comfortable and consumer-friendly AR glasses are just around the corner. One of the latest devices up in the air is Meta’s planned mixed reality headset currently called Cambria. This is a new product line separate from their successful Meta Quest 2.

However, the Cambria headset seems to be geared more toward wealthier audiences looking to get an early experience with the future of AR. Because of this, it seems that Cambria isn’t the magic bullet everyone was hoping for. However, it may be a step in the right direction.

Another important thing to watch out for is the evolution of Apple’s LiDAR scanner. Apple is one of the top companies predicted to introduce a consumer-focused AR headset or glasses in the future. In 2020, their advanced depth sensor was equipped on the iPad Pro, and later was equipped on the iPhone 12 Pro. The more that this technology and processing can be miniaturized, the more likely that we may see comfortable to wear ‘Apple Glasses’ in the future.

In addition to AR glasses, there are even more innovative devices that promise to take a prominent place among the augmented reality future trends. In June 2022, Mojo Vision Labs in Saratoga, California hosted the first demonstration of augmented reality smart contact lenses. Relying on eye tracking, communications and software, AR lenses integrate with user interface to enable an augmented reality experience. Mojo Lens has a custom-tuned accelerometer, gyroscope, and magnetometer that continuously track eye movements to ensure that AR images remain still as the eyes move.

Trend #7: AR in Marketing

There are a number of different applications for Augmented Reality in the marketing industry. For example, business cards are a popular and simple choice that can work with simple AR solutions. By adding interactivity to marketing material like a business card, you stand out from the competition and offer potential customers a whole new and exciting experience to get to know your company.

The MobiDev demo below shows how this business idea can be implemented using ARKit.

AR manuals are also a popular choice among businesses looking to provide their customers with more detailed and feature-rich instructions and documentation. AR not only allows for delivering information in an engaging way, but also significantly improves the user experience without forcing the buyer to spend a lot of effort to master one or another mechanism.

The MobiDev demo below presents a virtual user instruction for a coffee machine to show AR virtual manuals in action.

AR also has many opportunities for use in advertising. Web banner ads have decreased in popularity with users considerably over the years. Click through rate of banner ads have dropped from 0.72% in 2016 to 0.35% in 2019. One reason why this may be the case is those banner ads are disruptive to the content the user is trying to access. However, AR ads may provide more seamless access to content, obstructing content less. For example, with Facebook’s new augmented reality ads users can access AR experiences from their timeline with special ads with various capabilities. Some of these features include virtual try-on, placing virtual objects in their homes, and more.

Trend #8: Powering Indoor and Outdoor Navigation

In 2022, AR navigation has become more fluid and achievable than ever before. Most importantly, the rise of technologies like Bluetooth Low Energy (BLE) antennas, Wi-Fi RTT and ultra wideband (UWB) make indoor navigation much more viable than in previous years. One of the most useful applications of this technology is for displaying AR directions in large indoor locations like distribution centers, shopping malls, and airports.

Watch the demo below to find out how MobiDev implemented mobile AR for navigation of the corporate campus.

Something that shouldn’t be overlooked is this technology’s potential to be used by both consumer and business users. Just as a guest in a store may use AR indoor navigation to find the product they’re looking for, a distribution center worker may use it to find a particular item in their warehouse. Although comfortable and affordable glasses with AR capability aren’t quite here yet, the capacity for the business applications of AR in distribution centers, stores, and other sectors is there.

With indoor navigation, buy online pick up in store (BOPIS) services can be made much more efficient. Team members tasked to ‘pick’ the items in the store for order fulfillment can use AR directions to directly navigate to find the item as opposed to following coordinate directions to find the item. This eliminates time spent looking through many similar items and finding the correct aisle and section of the store. All the team member has to do is hold up their device and see the directions on the screen.

However, there are some limitations that need to be taken into account, such as items that have been misplaced around the store. If they have been moved by guests or incorrectly logged into the system, the team member might use AR navigation on their device to arrive at an empty spot on a shelf.

Trend #9: Healthcare and Augmented Reality

According to Deloitte Research, augmented reality and AI will transform the traditional healthcare business model by offering AR/MR-enabled hands-free solutions and IA-based diagnostic tools. For example, Microsoft Hololens 2 can provide information to the surgeon while allowing them to use both of their hands during the procedure.

With the continued restrictions associated with Covid-19, the use of augmented reality solutions is becoming increasingly important to address issues such as the complexity of remote patient support and the increased burden on hospitals. This includes both telesurgery solutions and mental health apps that are helping people to maintain psychological balance during these difficult times. For example, features such as drawing and annotating on the 3D screen can make communication between doctors and patients much easier and clearer. Remote assistance tools can also help clinicians support their patients while reducing downtime.

Combining with machine learning algorithms, AR technology can become an efficient option for disease detection. Back in 2020, Google announced the development of an AR-based microscope for the Department of Defense (DoD) to improve the accuracy of cancer diagnosis and treatment. Such a hybrid device uses a camera to capture images in real-time which are then processed using computer diagnostics to immediately display results and diagnose diseases at an early stage.

Trend #10: Augmented Reality Shopping Experiences

The onset of the COVID-19 pandemic called for numerous innovations that could help extend experiences to online shoppers. Augmented reality was one of the technologies that benefitted the most from this disruption. It resulted in an explosion of virtual try-on solutions.

Brands are actively adopting AR technology to improve the user experience when shopping online. For example, Dior has repeatedly launched AR shoes experiences allowing customers to virtually try on shoes before buying. Back in 2020, Dior teamed up with Snapchat to create such an initiative for the first time.

FRED Jewelry uses AR to let customers customize bracelets on the company website with a 3D configurator and try them on virtually.

FRED Jewelry Virtual Try-On Presented on Viva Tech 2022

SMART MIRRORS

As quarantine lockdowns have come to an end and brick-and-mortar stores have seen customers return, there is still an opportunity for AR to help with in-store experiences too. Smart mirrors are a great way to enrich the in-store experience and reduce the load on fitting rooms. Customers can walk up to smart mirrors and try on clothes in-store with advanced AR technologies not available on their smartphones.

Smart mirrors are also helpful in situations where certain sizes of clothes aren’t available in store and need to be shipped to customers. Smart mirrors and virtual fitting room technologies from home can help with these needs.

Trend #11: Augmented Reality in Manufacturing

Many AR applications are consumer-focused. However, AR has a lot of potential for use in industries like manufacturing. For example, worker training can be enhanced with AR experiences powered by CAD data. AR can also assist technicians through routine maintenance processes. AR applications can highlight elements of devices being worked on to guide technicians through the process at hand. This is generally more accessible through head-mounted solutions than through mobile applications.

In more simple applications, AR can help give workers more contextual information about objects in a factory when set up appropriately. By highlighting an object with a mobile device, a worker can learn more about it and if any action, such as maintenance, needs to be taken.

AR also has a promise for remote troubleshooting. Remote support agents can place virtual markers on the screen for workers to follow on the other end of the call. This can allow for more rich and valuable remote support in factory locations.

Trend #12: Augmented Reality in Automotive Industries

Augmented reality has a number of different applications that can be useful for the automotive industry. One of the more futuristic and interesting technologies emerging in this space is AR highlighting on-road objects through the use of a heads-up display (HUD). This can make drivers aware of hazards and GPS directions without requiring them to take their eyes off the road. AR is also in use for entertainment and information, such as 3D car manuals and other applications.

5G AND PARKING

One interesting application of AR in the automotive industry is for parking assistance. With the help of 5G connectivity, empty parking spaces can be highlighted on a driver’s heads-up display. This can also provide a great deal of data that can be useful for optimizing the layouts and operations of parking facilities like parking lots and garages.

WAKEUP APP: DRIVER AWARENESS ASSISTANCE

The WakeUp app developed by MobiDev also can be a great example of using augmented reality in the automotive industry. The objective of WakeUp is to help keep drivers awake by using ARKit facial recognition technology to detect when a driver’s eyes are closed or their head is tilted. If the eyes remain closed or head is tilted for too long, the device plays an alarm to help wake the driver up.

There is room for this technology to grow. For example, TrueDepth camera with its infrared sensing can help to perform head and eye tracking in complete darkness. Also, artificial intelligence could detect behaviors from a driver that indicate that they might become drowsy and alert the driver before it’s too late. These are the directions in which we are going to develop these products in the future.

The Future of Augmented Reality

The augmented reality market will continue to grow as the years go by, especially as technology becomes more and more accessible to consumers. With there being a significant growth in the focus on metaverse technologies, AR is the next step for many businesses. Those who are playing the long game may want to jump into this sector a bit early.

However, those looking to respond to more immediate growth and change may find better success in retail and mobile applications. AR-capable smartphones and tablets are everywhere and are great opportunities to advertise and extend conversion-driving experiences to users.

With the market expected to reach $97.76 billion in 2028, it’s clear that augmented reality is the future for many industries. That future will be determined by businesses that adapt to today’s challenges in new and innovative ways. Companies that offer rich AR experiences to their customers will be much better equipped to stand up alongside their competition.

Applying AI for Early Dementia Diagnosis and Prediction

Andrew Makarov — Mon, 18 Jul 2022 10:22:48 +0000

MobiDev would like to acknowledge and give its warmest thanks to the DementiaBank which made this work possible by providing the data set.

Mental illnesses and diseases that cause mental symptoms are somewhat difficult to diagnose due to the uneven nature of such symptoms. One such condition is dementia. While it’s impossible to cure dementia caused by degenerative diseases, early diagnostics help reduce symptom severity with the proper treatment, or slow down illness progression. Moreover, about 23% of dementia causes are believed to be reversible when diagnosed early.

Communicative and reasoning problems are some of the earliest indicators used to identify patients at risk of developing dementia. Applying AI for audio and speech processing significantly improves diagnostic opportunity for dementia and helps to spot early signs years before significant symptoms develop.

In this study, we’ll describe our experience creating a speech processing model that predicts dementia risk, including the pitfalls and challenges in speech classification tasks.

AI Speech Processing Techniques

Artificial intelligence offers a range of techniques to classify raw audio information, which often passes through pre-processing and annotation. In audio classification tasks we generally strive to improve the sound quality and clean up any present anomalies before training the model.

If we speak about classification tasks involving human speech, generally, there are two major types of audio processing techniques used for extracting meaningful information:

Automatic speech recognition or ASR is used to recognize or transcribe spoken words into a written form for further processing, feature extraction, and analysis.

Natural language processing or NLP, is a technique for understanding human speech in context by a computer. NLP models generally apply complex linguistic rules to derive meaningful information from sentences, determining syntactic and grammatical relations between words.

Pauses in speech can also be meaningful to the results of a task, and audio processing models can also distinguish between different sound classes like:

human voices
animal sounds
machine noises
ambient sounds

All of the different sounds above may be removed from the target audio files because they can worsen overall audio quality or impact model prediction.

HOW DOES AI SPEECH PROCESSING APPLY TO DEMENTIA DIAGNOSIS?

People with Alzheimer’s disease and dementia specifically have a certain number of communication conditions such as reasoning struggles, focusing problems, and memory loss. Impairment in cognition can be spotted during the neuropsychological testing performed on individuals.

If recorded on audio, these defects can be used as features for training a classification model that will find a difference between a healthy person, and an ill one. Since an AI model can process enormous amounts of data and maintain accuracy of its classification, the integration of this method into dementia screening can improve overall diagnostic accuracy.

Dementia-detection systems based on neural networks have two potential applications in healthcare:

Early dementia diagnostics. Using recordings of neuropsychological tests, patients can learn about the early signs of dementia long before brain cell damage occurs. Applying even phone recordings with test results appears to be an accessible and fast way to screen population compared to conventional appointments.

Tracking dementia progression. Dementia is a progressive condition, which means its symptoms tend to progress and manifest differently over time. Classification models for dementia detection can also be used to track changes in a patient’s mental condition and learn how the symptoms develop, or how treatment affects manifestation.

So now, let’s discuss how we can train the actual model, and what approaches appear most effective in classifying dementia.

How do you train AI to analyze dementia patterns?

The goal of this experiment was to detect as many sick people as possible out of the available data. For this, we needed a classification model that was able to extract features and find the differences between healthy and ill people.

The method used for dementia detection applies neural networks both for feature extraction and classification. Since audio data has a complex and continuous nature with multiple sonic layers, neural networks appear superior to traditional machine learning for feature extraction. In this research 2 types of models were used:

Speech-representation neural network which accounts for extracting speech features (embeddings), and
Classification model which learns patterns from feature-extractor output

In terms of data, recordings of Cookie Theft neuropsychological examination are used to train the model.

Image source: researchgate.net

In a nutshell, Cookie Theft is a graphic task that requires patients to describe the events happening in the picture. Since people suffering from early symptoms of dementia experience cognitive problems, they often fail to explain the scene in words, repeat thoughts, or lose the narrative chain. All of the mentioned symptoms can be spotted in recorded audio, and used as features for training classification models.

ANALYZING DATA

For the model training and evaluation we used a DementiaBank dataset consisting of 552 Cookie Theft recordings. The data represents people of different ages split into two groups: healthy, and those diagnosed with Alzheimer diseases — the most common cause of dementia. The DementiaBank dataset shows a balanced distribution of healthy and ill people, which means neural networks will consider both classes during the training procedure, without skewing to only one class.

The dataset contains samples with different length, loudness and noise level. The total length of the whole dataset equals 10 hours 42 min with an average audio length of 70 seconds. In the preparation phase, it was noted that the duration of the recordings of healthy people is overall shorter, which is logicall, since ill people struggle with completing the task.

However, relying just on the speech length doesn’t guarantee meaningful classification results. Since there can be people suffering from mild symptoms, or we can become biased for quick descriptors.

DATA PREPROCESSING

Before actual training, the obtained data has to go through a number of preparation procedures. Audio processing models are sensitive to the quality of recording, as well as omission of words in sentences. Poor quality data may worsen the prediction result, since a model may struggle to find a relationship between the information where a part of recording is corrupted.

Preprocessing sound entails cleaning any unnecessary noises, improving general audio quality, and annotating the required parts of an audio recording. The Dementia dataset initially has approximately 60% poor quality data included in it. We have tested both AI and non-AI approaches to normalize loudness level and reduce noises in recordings.

Huggingface MetricGan model was used to automatically improve audio quality, although the majority of the samples weren’t improved enough. Additionally, Python audio processing libraries and Audacity were used to further improve data quality.

For very poor quality audio, additional cycles of preprocessing may be required using different Python libraries, or audio mastering tools like Izotope RX. But, in our case, the aforementioned preprocessing steps dramatically increased data quality. During the preprocessing, samples with the poorest quality were deleted, accounting for 29 samples (29 min 50 sec length) which is only 4% of total dataset length.

APPROACHES TO SPEECH CLASSIFICATION

As you might remember, neural network models are used in conjunction to extract features and classify recordings. In speech classification tasks, there are generally two approaches:

Converting speech to text, and using text as an input for the classification model training.
Extracting high-level speech representations to conduct classification on them. This approach is an end-to-end solution, since audio data doesn’t require conversion into other formats.

In our research, we use both approaches to see how they differ in terms of classification accuracy.

Another important point is that all feature extractors were trained in two steps. On the first iteration, the model is pre-trained in a self-supervised way on pretext tasks such as language modeling (auxiliary task). In the second step, the model is fine-tuned on downstream tasks in a standard supervised way using human-labeled data.

The pretext task should force the model to encode the data to a meaningful representation that can be reused for fine-tuning later. For example, a speech model trained in a self-supervised way needs to learn about sound structure and characteristics to effectively predict the next audio unit. This speech knowledge can be re-used in a downstream task like converting speech into text.

Modeling

To evaluate the results of model classification, we’ll use a set of metrics that will help us determine the accuracy of the model output.

Recall evaluates the fraction of correctly classified audio records of all audio records in the dataset. In other words, recall shows the number of records our model classified as dementia.
Precision metric indicates how many of those records classified with dementia are actually true.

F1 Score was used as a metric to calculate harmonic mean out of recall and precision. The formula of metric calculation looks like this: F1 = 2*Recall*Precision / (Recall + Precision).

Additionally, as in the first approach when we converted audio to text, Word Error Rate is also used to calculate the number of substitutions, deletions, and insertions between the extracted text, and the target one.

APPROACH 1: TEXT-TO-SPEECH IN DEMENTIA CLASSIFICATION

For the first approach, two models were used as feature extractors: wav2vec 2.0 base and NEMO QuartzNet. While these models convert speech into text, and extract features from it, the HuggingFace BERT model performs the role of a classifier.

Extracted by wav2vec text appeared to be more accurate compared to QuartzNet output. But on the flipside, it took significantly longer for wav2vec 2.0 to process audio, which makes it less preferable for real-time tasks. In contrast, QuartzNet shows faster performance due to a lower number of parameters.

The next step was feeding the extracted text of both models into the BERT classifier for training. Eventually, the training logs showed that BERT wasn’t trained at all. This could possibly happen due to the following factors:

Converting audio speech into text basically means losing information about the pitch, pauses, and loudness. Once we extract the text, there is no way feature extractors can convey this information, while it’s meaningful to consider pauses during the dementia classification.
The second reason is that the BERT model uses predefined vocabulary to convert word sequences into tokens. Depending on the quality of recording, the model can lose the information it’s unable to recognize. This leads to omission of, for example, incorrect words that still make sense to the prediction results.

As long as this approach doesn’t seem to bring meaningful results, let’s proceed to the end-to-end processing approach and discuss the training results.

APPROACH 2: END-TO-END PROCESSING

Neural networks represent a stack of layers, where each of the layers is responsible for catching some information. In the early layers, models learn the information about raw sound units also called low-level audio features. These have no human-interpretable meaning. Deep layers represent more human-understandable features like words and phonemes.

End-to-end approach entails the use of speech features from intermediate layers. In this case, speech representation models (ALBERT or HuBERT) were used as feature extractors. Both feature extractors were used as a Transfer learning while classification models were fine-tuned. For these classification tasks we used two custom s3prl downstream models: an attention-based classifier that was trained on SNIPS dataset and a linear classifier that is trained on Fluent commands dataset, but eventually both models were fine-tuned using Dementia dataset.

Looking at inference results of the end-to-end solution, it’s claimed that using speech features, instead of text, with fine-tuned downsample models led to more meaningful results. Namely, the combination of HuBERT and an attention-based model shows the most concise result among all approaches. In this case, classifiers learned to catch relevant information that could help differentiate between healthy people and those with Dementia.

For the explicit description of what models and methods for fine-tuning were used, you can download the PDF of this article.

How to improve the results?

Given the two different approaches to dementia classification with AI, we can derive a couple of recommendations to improve the model output:

Use more data. Dementia can have different manifestations depending on the cause and patient age, as symptoms will basically vary from person to person. Obtaining more data samples with dementia speech representations allows us to train models on more diverse data, which can possibly result in more accurate classifications.

Improve preprocessing procedure. Besides the number of samples, data quality also matters. While we can’t correct the initial defects in speech or actual recording, using preprocessing can significantly improve audio quality. This will result in less meaningful information lost during the feature extraction and have a positive impact on the training.

Alter models. As an example of end-to-end processing, different upstream and downstream models show different accuracy. Trying different models in speech classification may result in improvement of classification accuracy.

As the test results show, applying neural networks to analyzing dementia audio recordings can generate accurate suggestions. Training neural networks for speech classification tasks is a complex exercise that requires data science expertise as well as audio processing knowledge.

ARKit vs ARCore: Comparison of Image Tracking Feature

Andrew Makarov — Wed, 27 May 2020 18:33:21 +0000

In my previous article, I covered using ARKit to develop indoor navigation applications. By using visual markers or the ARReferenceImage function within ARKit and ARCore’s Augmented Images, it’s possible to create powerful and flexible indoor navigation apps.

Now, I’ll go over each of these AR SDKs - Comparing the two, discussing their relative accuracies of the image tracking functionality, and describing how to use them in contexts other than just indoor navigation.

Both ARCore's Augmented Images and ARKit’s ARReferenceImage can identify 2-dimensional images in reality and superimpose a virtual image over those real-world images. They’re also both capable of real-time tracking of the movement of images.

As a developer, you can attach virtual images or other content to real-world surfaces, this opens a variety of AR use cases in marketing from virtual branding materials and business cards to advertising displays and posters outdoors.

What is better – ARKit or ARCore?

Over the course of 2019, ARKit was considerably more popular than ARCore, with ARKit being used on around 600 million devices versus 400 million for ARCore.

However, it’s worth noting that ARCore’s base of compatible Android devices grew by around 150 million devices from December of 2018 to May of 2019.

Github’s repository tells a similar tale, with more than 4,000 ARKit results versus more than 1,500 ARCore results as of May 2020.

Each of these platforms offers roughly comparable tools for tapping into motion sensors, monitoring lighting changes, and understanding environments, and both are Unity framework compatible.

ARCore has an advantage in the field of mapping. It can gather, parse, and store information about a 3-dimensional environment in a manner that allows for easy and simple re-access.

With ARKit, a relatively small quantity of similar information is retained, and a ‘sliding window’ of recent experience data is all that’s available to access.

ARCore creates a bigger mapping dataset, allowing for the possibility of increased stability and speed.

The face detection/tracking feature for iOS devices is quicker and more accurate than the comparable facial detection feature for Android devices due to TrueDepth Camera in iOS devices.

When it comes to recognition and augmentation of images, ARKit is superior to a significant degree. The below video draws a comparison between how the two SDKs function.

The user examines the renowned Mona Lisa painting, and the app imposes a virtual picture over the real picture. Note that the Mona Lisa can blink her eyes as the user taps the virtual image via the app.

Here, ARKit can surpass ARCore when it comes to delivering an immersive experience to app users. ARKit delivers a higher-quality image and can maintain image stability far better as the user moves their device around, which allows for using ARKit in non-obvious applications.

At this point, it seems far too early to choose a winner or loser between ARKit and ARCore. While it will be fascinating to see how the two platforms will develop and work with their respective strengths and weaknesses, right now it’s too close to call.

In all likelihood, businesses will have to develop solutions able to be used by devices running both platforms.

How to use ARKit for indoor positioning app development

Andrew Makarov — Thu, 21 May 2020 20:52:42 +0000

The process of developing an app for indoor spaces navigation has three stages:

1) Finding user’s position
2) Calculating the route
3) Rendering the route

Finding User’s Position

The ARKit 2.0 contains an ARReferenceImage function that can identify a 2-dimensional image within the real world and then use that image as a reference point for AR content.

Having scanned the 2-dimensional visual marker placed on a floor surface or a wall with the help of ARKit, then the app matches it with data on remote cloud to find the exact coordinates of it in the real world.

The ARReferenceImage object is made up of three data properties: an image, a name identifier, and the size of the image. However the name field can be used as unique identifier, which can then be linked up with a cloud-based coordinate set.

After the visual marker has been scanned, the resulting position of the user can be translated to 3-dimensional coordinates to represent our starting point.

Calculating the route

Provided that we can’t always reliably get a map of a given building with adequate scalability and picture quality, we must create a custom map using the Cartesian coordinate system and then align it with azimuth and geo coordinates using Google Maps or a similar solution.

An important notice: AR Ruler is a tool with bias issues, meaning traditional measuring tools are preferable.

Vector images allow for top-quality zooming with minimum transmitted data, resulting in excellent performance.

We’re then able to generate a graph by incorporating rooms and corridors with the placed visual markers locations.

Rendering the Route

Finally, it’s necessary to render the route itself. To begin, the image layer generated by the camera is overlaid with a 3-dimensional layer. As the route progresses around corners, we’d expect a wall to block the route, but we view the entire route. The resulting output is confusing and doesn’t look natural or aesthetically pleasing.

We have three potential solutions to this issue.

1) The first and simplest solution is to represent the route with an arrow, similar to the look of a compass. This works in certain contexts, but isn’t optimal for many use cases, including navigation apps.

2) Another solution is to only output the route within a fixed proximity from the user. This can be implemented fairly quickly and easily and does represent a solution to the issue.

3) The final, most progressive solution is to generate a low-poly building model with multiple 2-dimensional maps. This results in effectively clipping any part of the route that shouldn’t be visible. When it disappears around a corner, the route is clipped at that point. In a long stretch of straight hall, we’ll see the route until it vanishes around a corner. This type of route is easy for users to understand, and looks quite natural.

That's how the process looks. The video shows how ARKit-based app works for navigation inside the office building.

As a general rule, the two biggest factors in developing an AR indoor navigation app are the mapping and its overall complexity level.

It’s worth noting that this method has a technical limitation. It needs an uninterrupted session to function. To maintain proper accuracy, the user has to maintain an active camera even after scanning the starting marker, all the way to the final destination. It’s possible to mitigate this limitation by working with technologies like Wi-Fi RTT to leverage new methods of indoor positioning.

It's possible to do the same in ARCore with Augmented Images feature. Read my article about ARKit vs ARCore comparison for image detection and tracking.