VR/XR hardware changes faster than most production schedules. A Unity project can begin with controllers, add hand tracking, move to a different runtime, and encounter new display or tracking constraints before launch. My answer is not another layer of device checks. It is an architecture that separates user intention, interaction rules, hardware capabilities, and platform delivery. After 16 years in game development, including shipping Eagle Flight Arcade at Ubisoft Montreal in 2016 and working on modern XR client projects, I consider that separation one of the best investments an XR team can make.
Key Takeaways- Design around capabilities and user intentions, not headset model names.
- Keep OpenXR and toolkit APIs at the platform boundary.
- Give controllers, hands, and gaze appropriate interaction affordances.
- Treat frame timing, hardware testing, and telemetry as architectural concerns.
- Define fallbacks before optional XR features enter production.
Why does device-first XR architecture become expensive?
VR projects often begin with a deceptively reasonable question: which headset are we targeting? The dangerous next step is allowing that answer to shape every gameplay script. Code starts asking whether a particular controller exists, whether a specific runtime is active, or whether the current headset supports a named vendor feature. Those checks spread through tools, tutorials, UI, and game logic. When hardware requirements change, the project does not have one platform migration. It has dozens of small migrations hidden across scenes and prefabs.
I saw the importance of stable runtime behavior while working as a Gameplay Programmer on Eagle Flight Arcade at Ubisoft Montreal in 2016. That VR flight game shipped on PSVR, Oculus Rift, and HTC Vive. The devices were different, but the player-facing rules still needed to feel like one coherent game. The lesson I carried forward was not that every platform should behave identically. It was that hardware differences should enter the project through deliberate boundaries rather than leak into every feature.
OpenXR helps, but it is not a universal compatibility button. It standardizes communication with XR runtimes and defines interaction profiles, while hardware vendors can still expose optional extensions and different physical capabilities. One device may provide controllers and tracked hands. Another may add eye tracking. Tracking quality, refresh rates, permissions, boundary behavior, and system gestures can also differ.
I prefer capability-first design. Gameplay asks whether precise pointing, direct manipulation, haptics, or a confirmation action is available. A platform adapter decides how the active hardware provides it. This converts hardware changes from broad rewrites into contained adapter and presentation work. It also forces the team to decide what is essential, what is enhanced, and what requires a fallback before content depends on it.
Where should OpenXR end and game-specific code begin?
I treat OpenXR as infrastructure, not as the language of the game. The same applies to Unity's XR Interaction Toolkit. Both can provide valuable implementations, but a puzzle, tool, menu, or training workflow should not need to know which OpenXR interaction profile produced an action. If game code directly queries runtime paths or toolkit components, package decisions become gameplay dependencies.
A practical Unity project can use three layers. The platform layer owns OpenXR configuration, action bindings, tracking origins, runtime lifecycle events, and optional extensions. The interaction layer translates poses and actions into project concepts such as hover, select, grab, use, confirm, or cancel. The domain layer decides what those concepts mean. For example, the domain decides whether a component can be picked up while locked. It should not decide whether the request came from a trigger, pinch, gaze dwell, or test script.
This boundary also gives prefabs clearer responsibilities. A tracked controller prefab can contain pose drivers, render models, rays, haptics, and toolkit interactors. An interactable object can expose its states and rules without searching for a named controller in the scene. Scene code can then replace the controller presentation without rewriting the object. That matters when a prototype grows into a product with tutorials, accessibility settings, and multiple hardware configurations.
The boundary does not require wrapping every Unity API. Over-abstraction is another failure mode. I wrap the concepts likely to vary across hardware or tooling, while leaving stable engine concepts alone. Transform, collider, and animation code rarely need a custom facade. Input semantics, tracked-pose availability, haptic requests, and runtime capabilities usually do.
For teams that need help establishing this boundary, my Unity VR and MR development and performance optimization work focuses on production architecture as well as headset implementation. The goal is not abstraction for its own sake. It is keeping platform volatility away from the rules that make the experience valuable.
How can XR input represent intention instead of buttons?
The most durable input abstraction is an intention with temporal state. A button name is hardware vocabulary. An intention such as grab, use, confirm, or cancel is application vocabulary. The state must still preserve information such as whether the intention began this frame, remains active, or ended. Without that timing, gameplay scripts eventually return to polling raw controls.
It is important not to make the intentions too broad. A single generic select action can become ambiguous when the same hand can point at UI, grab an object, and operate a tool. I define a small vocabulary based on the experience's actual interaction model. I also keep context resolution outside the hardware adapter. The adapter reports intention, while interaction logic decides which eligible target receives it.
A minimal boundary can be expressed without referencing a controller type:
using UnityEngine;public enum XRIntent
{
Grab,
Use,
Confirm,
Cancel
}public readonly struct XRIntentState
{
public readonly bool IsHeld;
public readonly bool PressedThisFrame;
public readonly bool ReleasedThisFrame;public XRIntentState(bool isHeld, bool pressed, bool released) { IsHeld = isHeld; PressedThisFrame = pressed; ReleasedThisFrame = released; }}
public interface IXRIntentSource
{
XRIntentState Read(XRIntent intent);
}public sealed class XRToolDriver : MonoBehaviour
{
private IXRIntentSource input;public void Initialize(IXRIntentSource inputSource) { input = inputSource; } private void Update() { XRIntentState use = input.Read(XRIntent.Use); if (use.PressedThisFrame) BeginUsingTool(); else if (use.ReleasedThisFrame) StopUsingTool(); } private void BeginUsingTool() { } private void StopUsingTool() { }
}
The production implementation can read Unity Input System actions, hand gestures, simulated input, or recorded test data. The consumer does not change. Notice that the interface does not promise haptics or even a tracked hand. Those are separate capabilities. Mixing output features into input state makes fallback logic harder and encourages assumptions that every source has a physical controller.
When hand tracking or a new controller arrives, I add or update an intention source and then validate the interaction presentation. I do not rewrite every tool. This approach also makes input recording practical because a test can replay meaningful actions instead of reproducing vendor-specific button paths.
How should controllers, hands, and gaze share an interaction model?
Controllers, hands, and gaze can share semantics, but they should not be forced to share identical mechanics. A controller offers discrete buttons, predictable grip poses, and often haptic feedback. Hand tracking infers gestures from changing joint data and can lose confidence during occlusion. Gaze can indicate attention efficiently, but looking at something is not always consent to activate it. Treating these sources as interchangeable pointers produces fragile and sometimes exhausting interactions.
I begin by separating targeting, commitment, manipulation, and feedback. Targeting answers what the user may be addressing. Commitment confirms an intentional action. Manipulation describes continuous movement or adjustment. Feedback tells the user what the system understood. A controller ray might target an object, a trigger might commit, controller motion might manipulate it, and vibration might confirm contact. With hands, a pinch may commit while visual and audio feedback replace unavailable haptics. With gaze, I usually require a second signal or a carefully designed dwell rather than activating everything the user observes.
The shared layer should describe the result: an object became targeted, selected, grabbed, adjusted, or released. The source-specific layer should own gesture thresholds, pose stabilization, ray origin, dwell timing, and feedback options. This keeps application rules consistent without pretending that all modalities have the same reliability or ergonomics.
Capabilities can also change during a session. Controllers may sleep, hands may leave the tracked area, or permission for an optional feature may be denied. A robust interaction manager announces source availability and performs explicit transitions. It should clear stale hover states, release or safely preserve manipulated objects, and update tutorials or prompts. Silent switching is risky because the user may not know why an interaction stopped responding.
I also avoid collecting raw gaze or hand data by default. If a feature only needs a confirmed selection, store that event rather than a detailed biometric stream. Capability-first architecture makes this easier because domain analytics receive meaningful actions, not unrestricted sensor data.
Why must frame timing be part of XR architecture?
Performance is not a final optimization pass in VR. It affects which rendering features, physics rules, interaction techniques, and content densities are safe to build. At 72 Hz, a frame has about 13.9 milliseconds. At 90 Hz, it has about 11.1 milliseconds. At 120 Hz, it has about 8.3 milliseconds. That total includes CPU and GPU work, while the runtime also has presentation and tracking responsibilities. A feature that appears inexpensive in the Editor can still break frame pacing on a standalone headset.
I therefore make performance targets part of system interfaces. An interaction system should avoid uncontrolled physics queries per object, repeated scene-wide searches, and allocations during continuous hand or ray updates. Visual feedback should use pooled or persistent objects instead of creating effects every time hover changes. UI, outlines, transparent materials, shadows, and live previews all need costs that remain predictable when several interactors and targets are active.
The correct target is not merely a good average frame rate. Spikes matter because interaction events often trigger animation, audio, haptics, physics, and UI at the same moment. I profile frame-time distributions and investigate repeatable spikes around grabbing, releasing, opening menus, loading assets, and changing tracking modes. CPU and GPU timings must be considered separately because reducing scripts will not fix an expensive fragment shader.
Device testing should use release-like builds, the intended render path, and representative content. Thermal behavior also matters during longer standalone sessions. Dynamic resolution, foveation where supported, simplified effects, and quality tiers are useful controls, but they should protect an already disciplined frame budget rather than excuse an unbounded scene.
In 2025 and 2026, XR hardware continues to span standalone and PC-connected systems with different performance envelopes. A shared architecture should expose quality capabilities and chosen refresh targets without allowing every feature to invent its own device list.
How can teams test XR interactions without making the headset a bottleneck?
A headset is mandatory for validating an XR experience, but it should not be mandatory for every code change. If interaction rules consume intentions and poses through stable boundaries, much of the behavior can run in Edit Mode tests, Play Mode tests, or a desktop simulation. This shortens iteration and makes failures easier to reproduce than a verbal report about what happened inside a headset.
I divide testing into four layers. Domain tests verify rules such as whether an object is selectable, whether a two-step operation can be cancelled, and whether state survives an interruption. Interaction tests feed synthetic hover, selection, grab, and release sequences into components. Integration tests verify Unity Input System, XR toolkit, and OpenXR adapter configuration. Finally, device tests validate tracking, ergonomics, rendering, frame pacing, boundaries, permissions, and runtime lifecycle behavior.
Recorded intention sequences are particularly useful. A tester can capture a meaningful series of interactions, and the team can replay the sequence against later builds. This does not reproduce every tracking nuance, but it can catch state regressions such as an object remaining locked after input disappears. Synthetic tests should also cover ugly transitions: a source disconnecting while grabbing, tracking becoming unavailable during a confirmation, application focus changing, and the active modality switching.
The hardware matrix still needs discipline. I list supported runtime, device class, input modality, refresh configuration, and required optional features. I then identify which combinations receive full release coverage and which are unsupported. OpenXR does not remove this matrix. It makes the platform boundary more consistent, which should make the matrix easier to execute.
Real headset sessions should focus on qualities simulation cannot answer. Is a target readable at the expected distance? Can users discover the affordance? Does the interaction remain comfortable at different heights and orientations? Does tracking degrade around the intended physical motion? Automation protects logic, while human testing protects the embodied experience. A mature XR workflow needs both.
What should XR telemetry reveal after deployment?
XR bugs are often described through symptoms: an object would not grab, a hand disappeared, the view stuttered, or a menu stopped responding. Useful telemetry must provide context without collecting excessive sensor data. I want to know the application version, active XR runtime, broad device class, selected refresh rate, enabled capabilities, active interaction source, scene, and the semantic event that failed. Those fields are usually more actionable than a raw stream of poses.
Interaction state machines should emit structured events for transitions and rejected actions. If a grab request fails because the object is locked, report the reason. If a source becomes unavailable, record the transition and whether an active manipulation required recovery. If an adapter cannot bind a required intention, surface that during startup instead of waiting for a user to discover an inert control.
Performance telemetry should emphasize frame-time distributions and spikes rather than only average frames per second. Track CPU-bound and GPU-bound intervals where the platform exposes reliable measurements. Associate severe spikes with scene changes, quality changes, and high-level interactions. Avoid logging every frame to a remote service. Aggregate locally, sample responsibly, and send only what is needed to diagnose production behavior.
Privacy must be designed with the schema. Hand joints, room geometry, camera imagery, microphone input, and gaze can be sensitive. If a product needs to know that a user selected a button with gaze, the semantic selection event may be enough. Data collection should have a defined purpose, suitable consent, limited retention, and secure handling. The architecture should make the minimal path the easiest path.
Across freelance and client work, including Meta Spirit Sling and Loreal Viva Tech 2024, I have found that deployment context can differ as much as hardware. A public installation, standalone product, and guided demonstration may need different diagnostics. A stable event vocabulary lets telemetry adapt without embedding analytics calls throughout interaction code.
What should an XR team lock down before content production scales?
Before artists and designers produce large amounts of interactive content, I want a written platform contract. It does not need to predict every future headset. It needs to define the project's current promises and what happens when a capability is missing. Otherwise, each feature author makes a local assumption, and those assumptions become expensive dependencies.
- List supported runtimes, device classes, and interaction modalities.
- Define required capabilities and optional enhancements.
- Document fallback, interruption, and source-switching behavior.
- Set refresh targets and CPU and GPU frame budgets.
- Specify the device matrix required for release approval.
I also build one representative interaction slice before scaling content. It should include targeting, commitment, manipulation, feedback, interruption recovery, tutorial prompts, performance profiling, and structured diagnostics. This is not a visual showcase. It is evidence that the architecture survives a realistic workflow. Content templates and prefabs should come from that tested slice rather than from isolated prototypes.
Ownership matters. Someone must control OpenXR settings and platform adapters. Someone must define interaction semantics. Someone must maintain performance capture and hardware coverage. On a small team, one person may hold several roles, but the responsibilities should still be explicit. Package upgrades and runtime changes should pass through a branch with device validation instead of entering production as casual dependency updates.
My strongest recommendation is to decide what remains stable. Hardware adapters will change. Bindings and extensions will change. Rendering options will change. The domain meaning of selecting, using, confirming, and cancelling should change far less often. Put effort into that stable center, then keep platform volatility at the edges.
That is how I would build an XR project in 2026. OpenXR provides a valuable foundation, but durable production comes from capability boundaries, semantic interactions, measured frame timing, deliberate fallbacks, and repeatable testing. Those practices let a Unity project adopt new hardware without turning every new device into a rewrite.
Top comments (0)