Apple Could Someday Release an Apple Pencil-Like Controller for the Vision Pro

While Apple has made it clear that its upcoming Vision Pro headset doesn’t require any type of handheld controller, a recent Apple patent application indicates the Cupertino firm is looking at how to make an Apple Pencil-like accessory that will offer virtual brush tips only seen by the Vision Pro user.

While the Apple Pencil lineup has grown to three somewhat confusing models (Which iPad does this model work with? Which one charges via USB-C? Can this one charge itself from the iPad?), they’ve all been somewhat similar in size. However, the “Pencil” in this latest patent application’s illustrations resembles a much larger and chunkier stylus. This may reflect Apple’s desire not to make a writing implement accessory but aim for a handheld controller-like accessory instead.

“With one illustrative configuration… [the device] is a handheld controller having an elongated marker-shaped housing configured to be grasped within a user’s fingers,” says the newly-revealed patent application, called “Computer System with Handheld Controllers.”

“A handheld controller with a marker-shaped housing may have an elongated housing that spans across the width of a user’s hand and that can be held like a pen, pencil, marker, wand, or tool,” the patent application description continues.

Apple may be aiming to develop a stylus that can be used not only with the Vision Pro but also with other Apple devices. The patent application indicates that the device could be used as a controller for cellular phones, tablets, laptop computers, a wristwatch, and other electronic devices.

However, let’s get back to how the device would be used with the Vision Pro. The patent application indicates it could act as a marker with a variety of virtual tips.

“The head-mounted device or other device may have a display configured to display virtual content that is overlaid onto real-world content,” the patent says. “[A] handheld controller [could have] a tip portion onto which a computer-generated paint brush head is overlaid.”

The controller could feature an “inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc.”

The controller could also include wireless communication ability, which could offer “tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device.”

As is usual for patent applications like this, Apple’s application is rather broad, describing only a “handheld controller [which] may have a housing with an elongated shaft extending between first and second tip portions.”

The patent application is credited to five inventors: Stephen E Dey, Erin M Bosch, Yuhao Pan, lan P Colahan, and Christopher K Ewy.