DEV Community: Chrystian Vieyra

ARCore and ARKit Feature Sets Compared to Hamsters and DaVinci: How they see the world

Chrystian Vieyra — Thu, 10 Jan 2019 17:12:29 +0000

In the previous entry of this blog series, we looked into the market share of each AR platform, device compatibility, and perceived developer interest. In this article we will look into the main feature set that ARCore and ARKit provide. Overall, both platforms offer the same feature set, but there are differences in how they categorize and name these features.

ARCore identifies three main features: (1) motion tracking, (2) environmental understanding, and (3) light estimation. In contrast, ARKit refers to (1) tracking and (2) scene understanding, but the underlying technical aspects of each AR platform are essentially the same. Because Google and Apple provide fairly limited information to developers about how these features work, we provide some examples below.

Tracking Features

As the name of this feature suggests, smartphone AR technology tracks the position of the mobile device in space and builds a map of its environment through visual and inertial inputs. Google ARCore refers to this as motion tracking, while Apple ARKit refers to this simply as tracking.

ARCore and ARKit use the camera and a visual algorithm to identify feature points in the environment, such as sharp edges denoted by color and brightness contrast, shadows, and straight lines. Similar to a facial recognition system that can identify eyes, nose, a mouth, and the outline of a face based upon an understanding of how these objects are situated with respect to one another, the camera can get some sense of where two walls join together to make a corner of a room, or where a table edge is located.

Simply seeing a picture of a room is not enough to give it its three-dimensional quality, however. Unlike depth-sensing cameras available on some dedicated AR products (more on this later), the vast majority of smartphones use a simple camera that can be likened to the eye of a hamster. As is the case with most prey animals that need to be wary of their surroundings, hamsters do not have stereoscopic vision that gives them a sense of depth — each eye has an independent field of vision that ultimately allows them to have a wider field of view.

Using visuals alone, hamsters cannot immediately tell if the hungry predator is near or far, giving something of a flat, cartoonish effect to their perception of the world. However, hamsters aren’t totally incapacitated when it comes to understanding depth. Occasionally, you might notice your hamster pause, stand on its hind legs, and move its head from side to side. By doing this, the hamster slightly adjusts its view of the world and uses parallax cues to get a sense of what is near or far. When viewed from slightly different vantage points, objects that are nearer seem to shift their position with respect to a more static background.

Even with our — mostly — binocular, stereoscopic vision, humans can get a sense of how parallax adds to the perception of a simple camera. We say “mostly,” because a small but significant fraction of people have amblyopia. And, even this deviation from normal vision function doesn’t always present a handicap. Many artists, including Leonardo DaVinci, likely owe some of their visual genius to amblyopia because of their increased sensitivity to other visual cues like hue and intensity of color.

To illustrate how movement-induced parallax can help a simple camera get a sense of depth, you might recall the classic school activity in which you vertically hold out your index finger or a pencil, view it through one eye and then the other, and notice how your finger “jumps” with respect to the background. Instead of moving our head, this activity results in our seeing things from a different vantage point simply by ignoring input from one eye and then the other, as opposed to moving our heads about like a hamster or moving a smartphone camera about a room.

Image source: NASA

Just as we can correlate small head movements with the visual changes due to parallax, the smartphone combines inertial sensor data with the visual features.Essentially, the smartphone can generally notice how much it has been shifted in space with the use of an accelerometer, as well as how much it has been rotated with the use of a gyroscope, and correlate how much it has been moved to how much the visual features have shifted with respect to the background. (Curious to know how an accelerometer and gyroscope work? Check out our Sensor and Generator Info page to learn how each major sensor functions, or one of our previous blog posts that explains the foundational physics behind micro-electrical-mechanical technology.)

Image source: Google Developers

This technique of combining motion and visual information to navigate the world is known as visual odometry, and is a step up from dead reckoning that estimates change in position blindly. Using an accelerometer to determine changes in position involves double-integration of data, and can lead to cumulative positional errors can result in drift.

Drift is perhaps one of the biggest concerns for AR developers. Drift becomes apparent when it is clear that the AR experience is no longer anchored to the original position of the smartphone. In a sense, drift is a miscalibration between the augmented experience and reality itself. For example, in an ideal AR experience, you might place a virtual object on a table. If you turn and walk away with your phone, only to come back later, you would expect to find the virtual object in its original location. However, without strong visual anchors to counteract drift, such as too little visual data about a space or a room without significant visual characteristics, you might return to find that the object is in a different location — on a different part of the table, or not on the table at all. The strength of tracking in ARCore and ARKit is in the convergence of visual and motion inputs to provide a checks-and-balances system.

For best performance and to help provide visual anchors to the smartphone, it is best to translate (move) the device substantially before trying to interaction with AR. For reference, Google’s patented process for this is referred to as concurrent odometry mapping, while Apple’s patented process is referred to as visual inertial odometry, but the general idea is the same.

A special mention should be made about the new Apple iPhone X. Users might note that this popular phone does have a depth-sensing camera that uses an infrared pixel system. Although developers can access the camera data to incorporate into ARKit, it should be noted that this feature is only for the front-facing camera, which makes it good for little more than Face Recognition.

Understanding Features

The remaining features of ARCore and ARKit refer to making sense of the environment once it has been mapped. Google refers to this as environmental understanding and light estimation, while Apple combines these two into scene understanding.

A cursory look at how AR views its environment can be seen in the following two examples. In both approaches, the algorithm looks for planes. In the image below, the evenly-spaced white dots are an array that make up the plane that is the floor.

Separately, the algorithm looks for feature points, visual anchors within or outside of planes that provide additional information. In the case of the table, note that the feature points are not evenly spaced. Rather, they are only found on objects with visual contrast. There are no feature points on the white table cloth, but many can be found on the striped green placemats, and on the bowl and candle holder (with a number of feature points dotted along the their edges). As the smartphone is rotated in space, feature points appear and disappear as lighting changes.

As one might expect, both ARCore and ARKit algorithms are likely to work best in high-contrast, static environments where only the smartphone does the moving.

Lighting is not just used as an input for environmental understanding, however. Both ARCore and ARKit can use that information about the source(s) of environmental light from a static image selected from an AR session to enhance virtual images placed in the scene by providing them with realistic illumination and shadows.

Similar, but not Equal, Features

Although at the surface level these features appear similar, the reality is that they their precision cannot be compared until code is tested across devices. As mentioned in our previous post, Google has been extremely conservative in allowing Android devices to run AR, making ARKit much more accessible to people based upon shear numbers alone. Google has the challenge of creating ARCore that is compatible with the myriad types and qualities of sensors, potentially rendering ARCore very imprecise or unworkable with certain lower-end devices. These findings will guide our work in the development of AR software to collect and visualize magnetic fields sensed directly by the smartphone. In our next posting, we will begin to compare actual functionality of ARCore and ARKit with regard to drag and environmental understanding.

Contact the authors: Write to support@vieyrasoftware.net or visit www.vieyrasoftware.net

This work is funded by NSF Grant #1822728. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Comparing Google ARCore and Apple ARKit

Chrystian Vieyra — Tue, 01 Jan 2019 12:53:13 +0000

Choosing which platform to use to pursue exploratory work in augmented reality (AR) visualization techniques is critical. As many mobile developers only specialize on one platform, few cross-platform analyses exist for the relatively recent technological advancements like AR. In the following paragraphs, we present some of our own experiences with the investigation of these AR tools as part of a recently-awarded grant from the National Science Foundation to the American Modeling Teachers Associationand to Vieyra Software. We present a primer on ARKit and ARCore compatibilities with Apple and Android products. In future entries we’ll take a closer look at some of our own development on each platform, as well as inherent benefits and disadvantages in terms of performance and functionality of each as we work to use AR and real data to display magnetic fields.

In summer 2017, public AR frameworks were released openly to developers: ARKit for iOS on 5 June, 2017, and ARCore for Android on 29 August 2017. Linking the smartphone’s internal motion sensors and camera algorithms, these new frameworks allow developers to create AR worlds in which the smartphone has a 3-D environmental awareness. This allows users to not only display images on a 2-D background, but to interact with them in a 3-D environment without the use of cumbersome, pre-made visual targets. However, these bare-bones technology frameworks were released with extremely limited public documentation or sample applications off of which developers could base their work. Nearly a year following their release, these frameworks continue to be very poorly understood and underutilized by developers.

The yellow dots are anchor points for spacial awareness. The two red dots (on the mug and on the stack of napkin) are situated in space, with the distance between them noted in blue (in centimeters).

Who is developing with ARKit and ARCore?

A simple search on commit repositories shows that strong preference for ARKit by today’s developers. Despite the fact that both platforms were announced within a few months of each other, there is about five times more development and three times greater frequency of discussion on ARKit than ARCore.

Our grant work initially focused on Android ARCore as platform of choice because of a desire to create something that was as accessible as possible to users. However, seeing these numbers caused us to question why more developers weren’t investing in Android for AR.

Which devices are compatible with ARKit and ARCore?

As of 2018, nearly 85% of smartphones around the world use Android, a percentage that is expected to only grow in the coming years. However, consistent with a 2014 study of U.S. city maps that showed socioeconomic differences between the owners of Android versus iOS users, a huge portion of the billions of Android users are purchasing low-end devices that have significantly different capabilities than their higher cost counterparts. Unlike Apple, which has control over all models that use iOS, Android phones are manufactured by at least ten major companies, each of which can select for unique features and acquire disparate hardware components of their choice.

Portion of Android devices compatible with ARCore. Source: Google Developers

What does this mean for compatibility? Because of the hardware uniformity for iOS, all devices back to the iPhone SE and the iPad 5th generation are compatible with ARKit. Two notable estimates were made of the number of ARKit-compatible devices by the end of 2017–195 Million, which did not account for the release of the iPhone 8 or iPhone X, and 434 Million, an estimate released after the iPhone 8 had been revealed. Compare this with Google’s December 2017 estimate of ARCore-compatible devices of 100 Million, a comparative drop in the bucket with respect to Android’s massive global market share.

What variables influence the compatibility of devices with AR platforms?

Sensor Hardware: While all Apple devices have included the same basic sensor systems for years, Android devices are notoriously variable when it comes to hardware and sensors. ARCore requires a device with a back-facing camera, an accelerometer, and a gyroscope. While nearly all phones have the required camera and accelerometer, the gyroscope requirement removes compatibility with a large portion of low-end Android devices typically purchased by cost-conscious consumers and especially those in the developing world. Of all Android tablets, only the Galaxy Tab S4 is currently compatible. (Click here to see an expanded version of the table to the right for a complete list of ARCore-compatible Android devices).

Operating System: Compared to ARKit’s complete compatibility with devices released since September 2015 that are running iOS 11.0, ARCore supports only a limited subset of Android devices running Android 7 or newer, and users must download the app ARCore by Google, unlike Apple devices, which already include the app embedded within the operating system. Even these requirements for Android, however, aren’t hard-and-fast rules. Several devices require Android 8.0, and another subset of those require a specific patch (in addition to the download of the ARCore App), as is the case for the Sony Xperia XZ2.

App Requirements: For any ARCore-compatible device whose user has not already downloaded the ARCore by Google app, an attempt to download an app that uses ARCore capabilities, such as the popular IKEA Place app, notifies the user that ARCore by Google will also be downloaded.

Device Release: While all new iOS devices are ready for compatibility with ARKit, the same is not true for even the newest Android products. While this year’s Google smartphones, Pixel 2 and Pixel XL were released with ARCore fully installed, they only account for a minimal portion of the market share. Because Google does not control the manufacture of devices produced by other companies, most companies seek Android certification before release. In doing so, manufacturers work with Google to make sure that their hardware is compatible with Google Play apps, allowing for Google to modify their apps in advance to ensure that they work seamlessly once downloaded. Unfortunately, the ARCore by Google app appears to be excluded from this certification process, because a number of Android Certified devices have been released that are not compatible with ARCore, and cannot even be downloaded on them for testing by the user–Google effectively only permits the download of its ARCore app on devices that it has approved.

This inability to download the ARCore by Google app onto some of the newest and most popular Android devices as proven frustrating. For instance, the Galaxy S9 and Galaxy S9+ are one of Samsung’s most popular devices. Despite its release on March 16, 2018, Google did not permit the download of ARCore until early May, nearly two months after its release. Similarly, Sony’s XPERIA XZ2 was released on April 5, 2018, and wasn’t fully compatible until August 8, 2018, and even then only with the additionally required security patch.

How should one decide which platform to use to pursue development?

The answer depends upon your goals.

Functionality: While we haven’t yet explicitly identified the benefits and limitations to each AR software framework, our initial experimentation suggests that ARKit experiences significantly less positional lag than ARCore. (For example, when “placing” an inanimate AR object in the middle of a room, walking around the house, and returning with the expectation of finding the AR object at the same coordinates as it was placed, ARKit significantly outperformed our tests of ARCore on two different Android devices). The reasoning behind this might be that while all Apple-manufactured devices have sensors that have been calibrated in the same way, even the same Android manufacturers often pull from different sensor manufacturers with varying levels of sensor precision and accuracy. Indeed, the reason there have been delayed responses from Google in creating compatible ARCore software for new devices might be exactly because it is making adjustments to meet the hardware specs.

Accessibility: The goal of our recently-awarded National Science Foundation grant is to use modern 3-D mapping capabilities of phones, sensor data, and AR to build novel software for teaching and learning about fields that are highly accessible to device owners. Advancement in merging AR capabilities with sensor data has the potential to change the way educators, learners, and members of the workforce interact with their mobile devices, and, in turn, impact the way developers support the needs of education and workforce. We want to build on a platform that would eventually allow for wide-scale implementation of whatever we produced, especially educational resources for teachers and learners in underfunded schools and underdeveloped areas. However, the reality is that because so many Android devices are low-end, their users simply won’t have access to ARCore at this time. In terms of shear numbers, iOS and ARKit are much more likely to result in greater access to novel AR developments.

Although sensor-based software developers have long demonstrated a preference for Android because of the ease of access to sensor data and because of the multitude of sensor options available across devices, in the case of augmented reality, ARKit and Apple seem to win the day.

Contact the authors: Write to support@vieyrasoftware.net