Here Are the Gestures You’ll Use to Control Apple Vision Pro

Apple unveiled its much-anticipated Apple Vision Pro mixed-reality headset during its WWDC 2023 keynote address on Monday. If you watched the presentation, you likely noticed that Apple’s new “spatial computing” device does not use a physical keyboard, mouse, or joystick controller to input information and control the device’s features.

During a recent developer session, Apple demonstrated the built-in gestures that are used to interact with the Vision Pro:

• Tap – When you tap your thumb and index finger together you signal the headset that you wish to tap on an element in the virtual space you’re viewing on the headset display. Some users have begun calling this a pinch. Apple explains that this is the virtual equivalent of when you tap the screen of your iPhone.
• Double Tap – A double-tap of a finger performs a double-tap gesture. (Seems logical.)
• Pinch and Hold – A pinch and a hold gesture allows you to perform tasks like highlighting on-screen text.
• Pinch and Drag – By pinching and dragging you’ll be able to scroll and move around windows, both horizontally and vertically, the faster you move your hand, the faster you’ll scroll.
• Zoom – Zoom is a gesture that uses both hands, one of two gestures that require this. By pinching your fingers together and pulling your hands apart, you’ll zoom in on an element. Drag a window by the corner to adjust window sizes.
• Rotate – Rotate is the other two-handed gesture. Simply pinch your fingers together and rotate your hands. This allows you to manipulate objects in the virtual space seen on the headset’s screen.

Since there are multiple cameras in the Vision Pro, it can track your eye movements, detecting where you’re looking with precise accuracy. Your eye position helps to target the elements you want to interact with through hand gestures.

“Jazz hands” aren’t required with the Vision Pro, as hand gestures can be kept subtle instead of making grand gestures. Apple even tells users they can keep their hands in their lap, as it will prevent their arms from tiring by being held up in front of them. Other gestures are similarly minor; for instance, a tiny pinch gesture is the same as a tap. This is all due to the cameras’ precise tracking abilities.

Objects near and far can be easily selected and manipulated by simply reaching out and using your fingertips to interact with a virtual object.

Apple says the headset also supports other hand movements, including air typing. Unfortunately, the lucky few that have had a chance to spend time with a unit say they have not been able to try this feature out.

Gestures can also be “stacked” or used together. To create a drawing, simply look at a spot on the virtual canvas, select a brush with your hand, then draw in the air using another gesture.

Developers will have the ability to create custom gestures for use in their apps. Developers will need to ensure that their gestures are distinct from the system’s default gestures or from common hand gestures that people may use. Gestures should also be able to be repeated without hand strain.

The Vision Pro is also compatible with Bluetooth trackpads and mice, keyboards, and game controllers. Voice-based dictation and search tools are also part of the system.