Apple’s Headset Will Use Seamless ‘Extended Reality’ to Interact With Your iPhone and Mac

Apple is well-known for building tightly integrated ecosystems, so it should come as no surprise that the mixed-reality AR/VR headset it’s developing won’t be an island unto itself; it will be designed to work seamlessly with your iPhone and other Apple devices.

Features like Continuity and Handoff have become legendary, delivering an almost magical experience when working with multiple Apple devices together. Open an app on your iPhone or Apple Watch, and you’ll see an icon for it in your Mac’s dock, allowing you to transfer over whatever you’re doing without missing a beat. For instance, if you start an email on your iPhone and then decide that it’s too long to peck out on the touchscreen keyboard, you can transfer that draft to your MacBook with a single click and pick up right where you left off.

There’s also a plethora of other little touches that Apple has added over the years, from unlocking your Mac with your Apple Watch and handing off music from your iPhone to a HomePod to using your iPhone to enter text on your Apple TV and sharing files with AirDrop.

Hence, it’s a given that Apple’s upcoming mixed-reality headset will offer continuity-like features. However, a new European patent application is giving us some more insight into what Apple’s working on, and it looks like the company plans to take this seamlessness to a whole new level for its headset.

‘Extended Reality Systems’

The patent, which is titled Multi-Device Continuity for Use with Extended Reality Systems, describes a level of user experience that goes way beyond simply selecting an icon to transfer an app. Instead, it defines an “extended reality system” in which users will be able to interact with devices and move apps and data around simply by looking at things.

Apple provides some cool examples in the patent as to how it envisions this working, from visualizing data on pages that appear blank to everyone else to starting music playback by merely glancing at and gesturing toward a HomePod in the same room.

In one example, Apple describes how a user editing an email on their iPhone could quickly transfer their work in progress, put away the iPhone, and continue on the headset using an extended reality (XR) view that replicates a virtual screen suspended in front of them that they would interact with via hand and finger gestures to finish composing the email. This virtual display could be as large as the wearer’s field of view allows, with a full-sized virtual keyboard to replicate the typing experience. The user would simply move their fingers as if typing on a real keyboard, with the headset’s sophisticated camera system would pick up those movements and translating them into keystrokes.

In another example, Apple describes transferring music playback from an iPhone to a HomePod simply by looking at and gesturing toward the speaker in question — even from the other end of the room.

Three-dimensional information regarding the devices in the physical environment can be gathered by the XR device, and used to facilitate smooth and continuous transfer of control and/or content between the devices and/or the XR device.European Patent Office application 21718995.

The patent goes beyond HomePod speakers, describing how the headset could map out every device in the room. One example illustrates how it could overlay information in “accessory windows” that appear alongside an actual display, such as an iMac or a TV. This could be used to display contextual information that would be visible only to the wearer, such as identifying the actors in a movie or television show or displaying more personal and confidential information related to a file or email that’s open on a desktop computer.

Of course, as with most patent applications, it’s important to note that there are never any guarantees that these technologies will see the light of day, and we’re not holding our breath that all of this will arrive in Apple’s first-generation headset, which could be announced as soon as this year’s Worldwide Developer’s Conference (WWDC) in June. However, patents like these provide great insight into Apple’s thinking, and there’s little doubt the company is aiming to provide a user experience like this for its headset, providing it can work out all the kinks.

[The information provided in this article has NOT been confirmed by Apple and may be speculation. Provided details may not be factual. Take all rumors, tech or otherwise, with a grain of salt.]