Live Text in Photos
While Apple is rarely the first to introduce groundbreaking new features when it comes to its iOS releases, when it does introduce something, it almost always implements it in a groundbreaking new way. We saw this in how it approached copy and paste over a decade ago, and the latest example is its new text recognition feature in photos.
With iOS 15, it will not only be possible to search for text contained in pictures stored in your Photos app, but also effortlessly copy that text right out onto the clipboard and into other apps. This second part doesn’t just work in the Photos app either — you’ll be able to do it from just about any photo displayed on your iPhone, whether that’s in the Safari browser or using third-party apps.
Optical Character Recognition (OCR) is nothing new; many third-party services have offered the ability to search and extract text from photos for years, but what Apple has done here in iOS 15 is truly unique for two reasons.
Firstly, like everything else Apple has done with photo recognition, this is all happening right on the iPhone. Every other service uses cloud-based servers to do the heavy lifting of analyzing photos, meaning all of that text is stored in a database on someone else’s server. With Apple’s stance on privacy, it clearly doesn’t want any part of that, so the only place the data in your photos will be stored as text is on your iPhone.
Secondly, Apple is handling this in the easiest and most intuitive way we can possibly imagine. Copying text from a photo is literally as simple as dragging and dropping over the photo to select the text. There’s no need to switch into a separate mode or convert the full text of the photo into another app. You just select and copy like you would with a block of text anywhere else.
In addition, there’s no need to take a photo first if all you want to do is capture the text from something like a whiteboard or receipt. The live text feature is available directly in the Camera app, so you can point your iPhone camera at a block of text, and then immediately pull it into the clipboard without the need to go through the intermediate step of taking a photo that you’d likely just delete later anyway.
All of Apple’s smart data detectors are available for in-photo text as well, so if you have a picture that includes a phone number, you can simply tap on the number in the photo to place a call.
Lastly, it’s not just for text either. The new on-device intelligence can recognize a wide variety of objects in photos and provide more details with a tap. This includes things like looking up the breed of a dog, the type of a flower, or even identifying works of art and landmarks.