Apple Is Secretly Developing Revolutionary VR and AR Technology, Here’s Everything We Know so Far

With virtual/augmented reality revenue forecast to hit over $120 billion by 2020, it would be downright senseless to believe that Apple isn’t looking at ways to enter the space and capture a share of this booming market. Industry analysts predict that the company will pivot into the augmented reality space within the next two years, yet while other major players such as Google, Facebook, Microsoft and Sony have all started showing their cards, Apple remained characteristically secretive about their plans.

However, recent comments by Apple CEO Tim Cook do suggest that they might getting ready to move into the augmented/mixed reality arena: “Augmented reality will take some time to get right, but I do think that it’s profound,” explained Cook in a recent ABC News interview. According to him, AR enables people to be “very present” and have more productive conversations, amplified and enhanced by the virtual elements that seamlessly blend into their surrounding environment. Virtual Reality (VR), by contrast, isn’t as broad-based a technology in Cook’s view. “There’s no substitute for human contact, and so you want the technology to encourage that.”

There is some debate around the terminology of virtual technologies, but broadly speaking, virtual reality refers to fully immersive simulated experiences, while augmented reality refers to modifying – or adding elements to – your view of the real world.

If you believe you’ve never experienced augmented reality, just think of the face-morphing feature on Snapchat, or even the way that you can point your phone’s camera at a sign written in a foreign language and have Google translate that on the spot, not to mention the whole Pokémon Go craze.

A third emerging option in that space – in which Apple is rumored to have a special interest – is mixed reality, which blends much more advanced holographic images with your real-world, real-time environment. Microsoft’s HoloLens is currently the most advanced device available for creating and delivering those types of experiences.

There are many reasons why it would make sense for Apple to focus in that augmented/mixed reality space, not least because, as Tim Cook stated, it encourages interpersonal interaction rather than isolated – and isolating experiences. This is not to say that VR does not present social potential of course, but this connectivity must be deliberately designed into VR, whereas augmented/mixed reality naturally builds upon the environment – and the people around you, using the technology to seamlessly enhance it rather than building an external experience from scratch.

Another factor that makes AR/MR a natural fit for Apple, is the wearable devices that enable this type of technology, which tend to me much more, well, wearable. While VR headsets have so far all materialized as blocky and rather clunky-looking; the HoloLens (and Google Glass before it) are much closer to something you’d actually consider wearing out on the street without being embarrassed (or falling over on your face for that matter). And if anybody could pull off a design to elevate these displays into a must-have fashion accessory, it would most likely be Apple.

A look at the range of patents that Apple has filed over the years also provides further indication of how it is actively exploring this space. These include designs for a head-mounted display and a peripheral treatment for head-mounted displays. A 2008 patent application, for example, already described a personal display system, which simulated the experience of being in a movie theater. Similarly another goggle-like video headset allowed for a “personal media viewing experience” where users can watch movies and 3D content, while another head-mounted display system patent describes the use of a “laser engine” used to project images onto a clear glass display worn over the user’s eyes. 2010 and 2012 patents also described the use of motion sensors to create a 3D interface for iOS devices using augmented reality techniques. Apple described the interface as a “virtual room” navigated by manipulating the device orientation through built-in sensors and gestures.

Apple also filed a patent for an augmented reality feature in Maps, which would allow the app to get real-time estimations of the distance between users and various landmarks, as well as to overlay relevant information onto the landscape. There are solid and persistent rumors from credible sources such as Bloomberg that Apple is developing prototypes for smart glasses and a virtual reality headset, both of which are reportedly being worked on by secret teams assembled with high-profile strategic hires from companies such as Oculus and Magic Leap as well as HoloLens. Most recently the company has hired computer science professor Doug Bowman, who previously led the Virginia Tech’s Center for Human-Computer Interaction. He specializes in three-dimensional user interface design and has written a book on the subject covering 3D interfaces and the benefits of immersive virtual environments. He has extensive expertise with both virtual and augmented reality.

According to Business Insider, Apple is already working to integrate some AR functionality into the iPhone’s Camera app, yet it is not difficult to see that with the above patents and talent on board, it could bring smart glasses to market that connect wirelessly to your iPhone or Apple Watch, displaying images and overlaid information to the wearer in much the same way as Google Glass and the HoloLens do. Even if these rumors are true though, this would be unlikely to come to market before 2018 at the earliest. In the meantime, however, consumers are becoming hungry for immersive virtual experiences.

This opens up an interesting space for innovative technology start-ups to fill that gap with some out-of-the-box thinking. One such contender is London-based Zappar which has been developing augmented reality content and technology for over four years and recently launched a Kickstarter to develop an Apple-compatible and affordable mixed reality solution. Their crowdfunding campaign, which quickly attracted over 800 backers and far exceeded its $30,000 initial target, proposes to create a cardboard device for $30 that will generate MR experiences using devices such as the iPhone 6.

Zappar flaunts the company’s mixed reality
software on an iPhone 6 pictured above.

This video see-through approach is an interesting hybrid of a VR-style headset (in that it is fully enclosed) and MR interactivity, afforded by the fact that the phone’s in-built camera functionality is overlaid with a proprietary code that allows it to accurately map the images in relation to where the user is, and to add the holographic 3D elements to that in such a way as feels natural to the user.

Zappar Founder Simon Taylor explains that its technology relies on laying a series of markers – which he calls “pointcodes” around a room to allow the phone’s camera to orient itself accurately and map out the space in 3D. And as his company prepares to ship the ZapBox device by April next year, Taylor believes they can create a healthy ecosystem to produce a broad variety of mixed reality experiences for iPhone users: “Part of the reason we wanted to do the Kickstarter was to create this broad and inclusive ecosystem for mixed reality that is accessible to everyone,” he explains.

This is precisely the type of technology that could make sense for Apple to incorporate. That would also be in line with the long list of previous acquisitions by the company: These include Israeli start-up PrimeSense – makers of the motion-sensors originally used on Microsoft Kinect – as well as Emotient, Faceshift (the company behind the real-time motion capture technology in Star Wars VII: The Force Awakens) Metaio, who developed an app allowing for the quick and easy creation of AR experiences, and Flyby, which created an app that worked with Google’s 3D sensor-equipped Project Tango to attach messages to real-world objects.

Zappar is in a rather interesting position, in that it could replicate the VR democratization we’ve seen with Google Cardboard for mixed reality, allowing iPhone users to experience MR without investing in additional costly gear. It would certainly makes sense in the context of a longer-term strategy which would prepare the ground for the release of an Apple AR/MR device to compete with the HoloLens, since consumers who have tried such “entry level” experiences are much more likely to upgrade to higher-end ones.

It’s the sort of move that could help Apple catch up in this fast-evolving virtual space, as so far the majority of VR experiences simply to do not integrate with Apple products. Although devices like Google Cardboard are technically capable of working with the iPhone, most of the necessary apps simply aren’t available for the devices, and of course headsets like the Samsung Gear aren’t compatible at all. The Oculus Rift has similar issues, in that software must be specially created for it using the Oculus SDK. This is a problem for Macs because Apple does not prioritize high-end GPUs, and until they implement more powerful graphics cards, that is likely to remain an issue.

Oculus Founder Palmer Luckey went so far as saying that they have no plans to Support the Rift on Macs until Apple “Releases a Good Computer.” Ouch. Given Apple’s history of breakthrough innovation, however, we shouldn’t be surprised if they pull something extraordinary out of their hat soon. And that “one more thing” could very well turn out to be game-changer for AR.

How do you think Apple will change the augmented reality/virtual reality market?
Or do you think Apple is too late to the game?
Let us know in the comments below!

[The information provided in this article has NOT been confirmed by Apple and may be speculation. Provided details may not be factual. Take all rumors, tech or otherwise, with a grain of salt.]