Shoe Shopping? Measure Your Size, Get the Perfect Fit with the iPhone’s TrueDepth Camera and This Cool App

As the capabilities of the iPhone camera system keeps improving, developers are finding even more practical uses for features like augmented reality and facial recognition, and now an innovative Canadian startup has found a way to use it to help you buy shoes.

As reported by iPhone in Canada, a small company named Xesto that came out of the University of Toronto’s Early Stage Technologies program (UTEST) has figured out a way to use the iPhone’s camera system to accurately measure the size of a person’s feet.

The developers worked closely with researchers in the University’s Faculty of Applied Science and Engineering, specifically within the division focused on electrical and computer engineering, and what’s particularly fascinating is that their solution relies solely on the facial recognition system of the iPhone’s TrueDepth camera, and not the augmented reality frameworks that are more commonly used for such things.

While it would certainly be possible to leverage the LiDAR Scanner, which can obviously take very accurate measurements, the folks at Xesto obviously wanted to make sure their solution worked on as many iPhone models as possible, and of course, the TrueDepth camera already includes the 3D depth mapping system needed to capture accurate measurements. Hence, Xesto’s app can be used on any iPhone with a TrueDepth camera, going back to the 2017 iPhone X.

In fact, it was the iPhone X that first inspired the project, which shifted direction after the researchers became inspired by the appearance of the TrueDepth camera and the possibilities unlocked by it. Before that Xesto, which is named for the Galician word for “gesture,” had been focused on using gestures and hand tracking to interact with computers.

How It Works

It’s a timely development considering how much harder it is right now to buy shoes in-person in the midst of a global health pandemic, and the app, Xesto Fit, has actually been available on the App Store since December, as the team wanted to get it out in time for the holiday shopping season. However, it largely flew under the radar until it was featured this week in U of T News, the University of Toronto’s campus-wide newsletter.

After downloading the free app, users simply take four or five different pictures of each of their feet using the front TrueDepth camera, and then press a button to get an accurate size. As master’s student and team researcher Jeffrey Qiu notes, “That’s the magic behind it — the user doesn’t need to know what mathematical transformations are reconstructing the foot from point cloud frames.” It just works.

Xesto has already been granted one patent for its method, and has another one pending, but notes that using “point clouds” — essentially “3D pixels” — the team’s algorithms can pinpoint foot sizes down to an accuracy of 1.5mm.

However, Xesto Fit doesn’t stop at simply offering up a measurement; it also cross-references it with the siding guidelines of over 150 different shoe brands for an even more finely tuned measurement of the perfect fit to make shoe size recommendations in different brands.

Further, users can also interact with the 3D models of their feet in a photorealistic view or simply as point clouds, and share their foot profiles with friends and family to help with gift purchasing, so you may never need to worry about finding the wrong size of shoes under the Christmas tree again.

The app is also isn’t just limited to the iPhone — a native iPad version is also available that can be used on the TrueDepth-equipped 2018 and 2020 iPad Pro models.

How It Started

For Xesto’s team, the project is an area of deep interest and passion, with the developers noting that they’ve been interested in this kind of technology since their undergraduate days, and they’ve also been using the same “point cloud reconstruction” methods to study applications in health care, self-driving cars, and even space exploration. However, Xesto Fit has given them an opportunity to step out of the theoretical and into a more practical world that can benefit many more people on a daily basis.

Comment

What really stood out for me at Xesto was that they were using something so innovative and new at a level that could impact the everyday consumer. I didn’t think of using it for anything other than self-driving cars. That blew me away.

Najah Hassan, researcher and MASc. Candidate, University of Toronto

Xesto was originally co-founded by Sophie Howe, a finance and economics major, and Afiny Akdemir who is working toward a doctorate in applied mathematics, who assembled their core team via their U of T affiliations, specifically Professor Deepa Kundur, the Chair of the University’s Department of Electrical and Computer Engineering, who connected them with graduate student researchers such as Qiu and Hassan.