DEV Community: Vladyslav Kaplun

The future of augmented and virtual reality games

Vladyslav Kaplun — Thu, 04 May 2023 14:15:14 +0000

Virtual reality (VR) and augmented reality (AR) have been on the scene for quite a while. These technologies have been in development for several decades. However, recent evolution in hardware and software triggered a renewed interest in both AR and VR. VR turned from technological high-end to a viable consumer technology with the release of Oculus Rift headset in 2016. Rift was the first VR product to offer a truly immersive experience. Soon other headsets were launched such as HTC Vive, PlayStation VR, and Samsung Gear VR. As for the AR, Microsoft HoloLens headset in 2016 was a major technology breakthrough. Software AR platforms, such as Google's ARCore and Apple's ARKit were a more recent development, bringing AR technology to a much wider audience through mobile devices.

Gaming potential of AR/VR

AR and VR have already revolutionised the gaming industry. AR/VR games allow players to physically manipulate virtual objects or environments, bringing the sense of control to levels previously unmatched. Players can have new ways of real-time interaction with each other, even if they are physically located in different parts of the world. Such games can provide new opportunities for socialisation, as players can interact with each other in a shared virtual space. There are also opportunities for AR/VR to enhance the accessibility of gaming, enabling players with physical disabilities to engage with games with alternative means of control and interaction.

The first AR game that became a worldwide bestseller and even a cultural phenomenon, was Pokémon GO, released in 2016. Still very popular today, it allows players to explore the real world and catch Pokémon using their smartphones’ camera and GPS. By using AR technology to overlay digital creatures onto the real world, Pokémon GO created a new type of gaming experience that was both immersive and social. Another well-known example is The Walking Dead: Our World. Similar to Pokémon GO, it is a location-based AR game that overlays digital content onto the real world. In this case, however, the game is focused on battling zombies and saving survivors in a post-apocalyptic virtual setup. Players can team up with friends to engage in the gameplay together.

There are also several notable VR products that contributed to changing the gaming scene. For instance, have a look at Beat Saber, a music rhythm game in which players wield two virtual lightsabers and slice through blocks along with the beat of the music. The game has received widespread critical acclaim and has become a favourite among VR enthusiasts. Another VR groundbreaker is Lone Echo, a space adventure game that creates a highly immersive and realistic zero-gravity environment. It is powered by an advanced physics engine and has intuitive controls that allow players to move and interact with objects in a way that feels natural and intuitive.

AR/VR game dev challenges

Developing an AR/VR game can pose a number of unique challenges on top of the usual ones that any tech company faces:

The major challenge for an AR/VR game developer is designing a user interface that would be intuitive and easy to use. In a traditional game, players typically use a controller or a keyboard. AR/VR games often employ gestures or hand movements instead. This means that developers need to carefully design their interfaces to ensure they are easy to learn and to use, but also provide players with enough control and flexibility to engage with the game.

Another big challenge is mitigating motion sickness. Due to the immersive nature of AR/VR, it can cause some people to feel sick or even experience nausea. This can be caused by numerous factors such as lag, low frame rate, or incorrect calibration of the player's head tracking. To bring these problems to a minimum, developers need to carefully optimise their games to ensure smooth and consistent graphics, and also provide options for players to adjust settings such as field of view or movement speed.

Hardware compatibility is another challenge that developers face when creating AR/VR games. Developers need to make sure their games are compatible with a wide range of highly specialised devices, like VR headsets or AR-enabled smartphones. This can be particularly challenging as new devices are constantly being released offering different specifications and capabilities.

Finally, developing AR/VR games is definitely more resource-intensive compared to traditional game dev. Providing convincingly high levels of immersion and interactivity, such games require more complex and detailed game worlds, as well as graphics and sound design of much higher quality. Also, in-game physics models need to be more sophisticated and fast at the same time. These imperatives mean that developers need to invest more time and resources into creating their games, driving dev costs higher.

AR/VR future: gaming and beyond

In terms of software, the newest developments in AR/VR will be enabled by cutting-edge technology in computer vision, machine learning, and natural language processing. Hardware-wise, the next milestone of AR/VR technology is likely to be about making these technologies more accessible and user-friendly. This could include lighter and more comfortable headsets. As devices like Oculus Quest 2 become more affordable, we can expect to see more people adopting this technology and experimenting with new ways to use it. Also, we may see the development of new accessories such as haptic feedback gloves or full-body tracking systems that would enable experiences that we can hardly imagine today. Finally, the increasing availability of faster 5G networks will also support the performance and functionality of AR/VR technology, enabling more seamless and immersive experiences as more and more data can be pumped through wireless communication channels.

There is also the potential for AR/VR gaming to expand into new areas beyond entertainment. In the field of education AR/VR can be used to create immersive learning courses. By using virtual environments to simulate real-world scenarios, students can gain experience and practice skills in a safe and controlled environment. This technology can also be used to make learning more accessible, as it allows students to explore topics that may be difficult or impossible to experience in real life. Medical students can use AR/VR simulations to practise surgeries and gain experience before performing procedures on real patients. Also, AR/VR can be used to create immersive therapy experiences for patients, such as exposure therapy for individuals with phobias or PTSD.

But, like all technology, AR/VR benefits cannot come without risks alongside. On top of the usual concerns about privacy and data security, typical for the entire tech industry, there is a new menace: the potential for addiction and blurring of the line between reality and virtual reality. Also, high costs of the AR/VR may deepen inequality, as it may be less accessible to those with lower socioeconomic status. As the technology continues to develop, it will be important to carefully consider its broader implications and ensure that it is used in a way that benefits society as a whole

A Comprehensive guide to build a cross-platform application by Bazel

Vladyslav Kaplun — Thu, 17 Nov 2022 09:09:47 +0000

Intro

New to Bazel? Or maybe you were about to use it for your cross-platform application? You’re in the right place.

You can find a decent amount of tutorials to use Bazel in your development. Most of them are either outdated or just don't work with the latest version. I was surprised that a project that size lacks a complete tutorial to build a cross-platform application. Let's fill that niche

What you'll learn

In this quick guide, we are going to use Bazel to build multiple applications which share a native library. We will be targeting 4 platforms: iOS, macOS, Android and Web.
In the end you should have a solid basement to build your first cross-platform application (or move an existing project to Bazel). I listed the project on Github - I encourage you to clone it and experiment.

Pre-requisite

This guide assumes you already have some basic knowledge of how Bazel works and what it is, how WORKPLACE and BUILD files are structured. I'll be focusing on BUILD files contents describing all the details, without a deep dive into platforms internals, JNI, Cocoa bridging or how Emscripten compilation works.

Before you begin

You'll need these tools installed on your system to compile & run this guide. My goal was to use the latest available version for each them, but I assume their older versions should also work fine

bazel 5.3.2
Android SDK
- Android API 32
- Android NDK 21. NDK v21 is currently the latest supported version
- Android Build-Tools
- Android Emulator
- Android Platform-Tools
- Android Studio. This is not a hard dependency. Though, Android Studio significantly decreases complexity of installing Android SDK and Android Emulator set up
Xcode 14

Project architecture

As a shared dependency, we will create a small C++ library which will be used by every application. We will also create a bridge for each of the platforms so the native code could be imported. For the web app, we will use Emscripten to convert the native library to Javascript. Projects files were generated by Xcode and Android Studio and used without any modifications

Project file-tree:

├── WORKSPACE
├── common
│   ├── BUILD
│   ├── android_bridge
│   ├── cocoa_bridge
│   ├── library
│   └── web_bridge
├── project.android
│   ├── BUILD
├── project.ios
│   ├── BUILD
├── project.mac
│   ├── BUILD
└── project.web
    ├── BUILD

Build with Bazel

Common library

Just like the first step every developer does, we will be displaying the greeting. The library will contain a single class with a method which returns the message

`common/library/source.cpp`

std::string Library::sayHello() {
    return "Hello, World!";
}

The BUCK file just lists the sources and the name:

`common/BUILD`

cc_library(
    name = "library",
    srcs = [":library/source.cpp"],
    hdrs = [":library/header.hpp"],
)

Cocoa bridge

Swift can not consume a native library directly, and needs a bridge, which is basically an Obj-C wrapper. Our wrapper class contains a single static method which returns NSString instance:

`common/cocoa_bridge/NativeBridge.mm`

@implementation NativeBridge

+ (NSString*) sayHello {
  const std::string hello = Library::sayHello();
  return [NSString stringWithCString: hello.c_str() 
                            encoding: [NSString defaultCStringEncoding]];
}

@end

Bridge definition is similar to library's BUILD:

`common/BUILD`

objc_library(
    name = "cocoa_native_bridge",
    srcs = [":cocoa_bridge/NativeBridge.mm"],
    hdrs = [":cocoa_bridge/NativeBridge.h"],
    visibility = ["//visibility:public"],
    deps = [":library"],
)

iOS

To display the message, I have created a label and imported it to the ViewController via IBOutlet. We just need to fill the label with our text:

`project.ios/ViewController.swift`

class ViewController: UIViewController {
  @IBOutlet weak var label: UILabel!

  override func viewDidLoad() {
    super.viewDidLoad()

    label.text = NativeBridge.sayHello()
  }
}

By default, Bazel does not know how to build an iOS application, so we need to install rules which define how to build it:

`WORKSPACE`

git_repository(
    name = "build_bazel_rules_apple",
    remote = "https://github.com/bazelbuild/rules_apple.git",
    tag = "1.1.3",
)

git_repository(
    name = "build_bazel_rules_swift",
    remote = "https://github.com/bazelbuild/rules_swift.git",
    tag = "1.2.0",
)

load("@build_bazel_rules_swift//swift:repositories.bzl", "swift_rules_dependencies")

swift_rules_dependencies()

Now, using the imported rules, we can define the iOS target:

`project.ios/BUILD`

ios_application(
    name = "bundle",
    app_icons = [":Assets.xcassets/AppIcon.appiconset/Contents.json"],
    bundle_id = "bazel.xplatform.awesome",
    families = [
        "iphone",
        "ipad",
    ],
    infoplists = [":Info.plist"],
    launch_storyboard = ":Base.lproj/LaunchScreen.storyboard",
    minimum_os_version = "13.0",
    version = ":version",
    deps = [":sources"],
)

apple_bundle_version(
    name = "version",
    build_version = "1.0",
    short_version_string = "1.0",
)

swift_library(
    name = "sources",
    srcs = [
        ":AppDelegate.swift",
        ":SceneDelegate.swift",
        ":ViewController.swift",
    ],
    copts = [
        "-import-objc-header",
        "common/cocoa_bridge/Native-Bridging-Header.h",
    ],
    data = [
        ":Base.lproj/Main.storyboard",
    ],
    module_name = "sources",
    deps = ["//common:cocoa_native_bridge"],
)

ios_application does not allow us to include sources directly, so we need an additional target swift_library.

Open the terminal.app and execute this in the root directory:

bazel run //project.ios:bundle

If everything goes as expected, you should see something like:

macOS

The same steps should be applied inside the ViewController for macOS :

`project.mac/ViewController.swift`

class ViewController: NSViewController {
  @IBOutlet weak var label: NSTextField!

  override func viewDidLoad() {
    super.viewDidLoad()

    label.stringValue = NativeBridge.sayHello()
  }
}

Xcode does not create an Info.plist for generated projects, which is a hard dependency for Bazel.
Let's create one:

`project.mac/Info.plist`

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>NSMainStoryboardFile</key>
    <string>Main</string>
</dict>
</plist>

The target definition is very similar to the iOS target:

`project.mac/BUILD`

macos_application(
    name = "bundle",
    app_icons = [":Assets.xcassets/AppIcon.appiconset/Contents.json"],
    bundle_id = "bazel.xplatform.awesome",
    entitlements = ":bundle.entitlements",
    infoplists = [":Info.plist"],
    minimum_os_version = "12.0",
    version = ":version",
    deps = [":sources"],
)

apple_bundle_version(
    name = "version",
    build_version = "1.0",
    short_version_string = "1.0",
)

swift_library(
    name = "sources",
    srcs = [
        ":AppDelegate.swift",
        ":ViewController.swift",
    ],
    copts = [
        "-import-objc-header",
        "common/cocoa_bridge/Native-Bridging-Header.h",
    ],
    data = [
        ":Base.lproj/Main.storyboard",
    ],
    module_name = "sources",
    deps = ["//common:cocoa_native_bridge"],
)

One more application is ready:

bazel run //project.mac:bundle

A similar window should appear on your screen:

JNI bridge

Just like Swift, Java can not import native code directly.
Our JNI bridge will be split into 2 targets:

native wrapper which exports native methods to JVM
Java class which "consumes" those methods

Native method export:

`common/android_bridge/jni.cpp`

extern "C" JNIEXPORT jstring JNICALL Java_library_NativeBridge_sayHello(JNIEnv* const env, const jclass clazz) {
  const std::string res = Library::sayHello();
  jstring result = env->NewStringUTF(res.c_str());
  return result;
}

Java class which imports the exported code:

`common/android_bridge/java/library/NativeBridge.java`

public class NativeBridge {
  public static native String sayHello();
}

Time to show Bazel how to build an Android application:

`WORKSPACE`

git_repository(
    name = "build_bazel_rules_android",
    remote = "https://github.com/bazelbuild/rules_android.git",
    tag = "v0.1.1",
)

android_sdk_repository(
    name = "androidsdk",
    path = "<path to installed android sdk>",
)

android_ndk_repository(
    name = "androidndk",
    api_level = 21,
    path = "<path to installed android ndk>",
)

The final target to export the bridge:

`common/BUILD`

android_library(
    name = "android_native_bridge",
    srcs = [":android_bridge/java/library/NativeBridge.java"],
    visibility = ["//visibility:public"],
    deps = [":android_native"],
)

cc_library(
    name = "android_native",
    srcs = [":android_bridge/jni.cpp"],
    linkopts = ["-ldl"],
    deps = [":library"],
    alwayslink = True,
)

The fun part worth to mention:

linkopts = ["-ldl"],
alwayslink = True,

These attributes are not listed in any Bazel tutorial I found! If you omit those, linker will simply drop the symbols, making the library empty. This was raised to Bazel community and eventually should be fixed in a new rule.

Android

Time to show the message from the Activity:

`project.android/java/bazel/xplatform/awesome/MainActivity.java`

public class MainActivity extends AppCompatActivity {

    static {
        System.loadLibrary("bundle");
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        TextView text = (TextView)findViewById(R.id.text_view);
        text.setText(NativeBridge.sayHello());        
    }
}

We will need an extra rule to import maven dependencies:

`WORKSPACE`

git_repository(
    name = "rules_jvm_external",
    remote = "https://github.com/bazelbuild/rules_jvm_external.git",
    tag = "4.2",
)

load("@rules_jvm_external//:defs.bzl", "maven_install")

maven_install(
    artifacts = [
        "androidx.appcompat:appcompat:1.3.0",
        "com.google.android.material:material:1.4.0",
        "androidx.constraintlayout:constraintlayout:2.0.4",
    ],
    repositories = [
        "https://maven.google.com",
    ],
)

The Android target list the sources and adds maven libraries as a dependency:

`project.android/BUILD`

android_binary(
    name = "bundle",
    srcs = [":java/bazel/xplatform/awesome/MainActivity.java"],
    custom_package = "bazel.xplatform.awesome",
    manifest = ":AndroidManifest.xml",
    manifest_values = {
        "minSdkVersion": "15",
        "targetSdkVersion": "32",
    },
    resource_files = glob(
        ["res/**/*"],
        ["**/.DS_Store"],
    ),
    deps = [
        "//common:android_native_bridge",
        "@maven//:androidx_appcompat_appcompat",
        "@maven//:androidx_constraintlayout_constraintlayout",
        "@maven//:com_google_android_material_material",
    ],
)

To run the application, you'll need a launched emulator or a device connected to your Mac/PC. For simplicity, I have listed every available architecture:

bazel mobile-install //project.android:bundle --start_app --fat_apk_cpu=armeabi-v7a,arm64-v8a,x86,x86_64

An emulator version:

Web bridge

We need to bind the library before it can be converted to JavaScript:

`common/web_bridge/embind.cpp`

class NativeBridge {
public:
  static std::string sayHello() {
    return Library::sayHello();
  }
};

EMSCRIPTEN_BINDINGS(xplatform_awesome) {
  emscripten::class_<NativeBridge>("NativeBridge")
      .class_function("sayHello", &NativeBridge::sayHello);
}

This rule will download required Emscripten version:

git_repository(
    name = "emsdk",
    remote = "https://github.com/emscripten-core/emsdk.git",
    strip_prefix = "bazel",
    tag = "3.1.25",
)

load("@emsdk//:deps.bzl", "deps")

deps()

load("@emsdk//:emscripten_deps.bzl", "emscripten_deps")

emscripten_deps()

Finally, the target is ready for the web build:

`common/BUILD`

cc_binary(
    name = "web_native_bridge",
    srcs = [":web_bridge/embind.cpp"],
    linkopts = ["--bind", "-sSINGLE_FILE"],
    visibility = ["//visibility:public"],
    deps = [":library"],
)

Web

I have created an HTML page to load the wasm code:

`project.web/index.html`

<!doctype html>
<html>
<script>
  var Module = {
    onRuntimeInitialized: function () {
      const text = Module.NativeBridge.sayHello();
      document.getElementById('text_view').innerText = text;
    }
  };
</script>
<script src="web_native_bridge.js"></script>

<body>
  <p id="text_view" style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%);">Loading WASM...
  </p>
</body>

</html>

The web target converts C++ code to JavaScript:

`project.web/BUILD`

wasm_cc_binary(
    name = "bundle",
    cc_target = "//common:web_native_bridge",
)

Creating and running a web container is out of bounds for this guide, so I've created a simple script which copies the html file and opens it in a browser:

bazel build //project.web:bundle && bash project.web/copy_and_run_html.sh

Results

Bazel is an amazing tool which can solve almost any assigned task. Using bazel rules we can onboard any possible build dependency.
With this guide, you have a solid example of how to include shared logic in your cross-platform application!