Upanishad SharmaEven though the WWDC 2023 devcon was rife with various announcements, the main point of focus was undoubtedly the Apple Vision Pro headset, the company’s state of the art next generation mixed reality headset. The ‘spatial computer‘ is jam-packed with compelling new tech that takes the MR experience to the next level. While we have already given you an info dump above, today we discuss some of the best Apple Vision Pro features. So sit back and experience what the future has in store for us.
1. 3D Layered Responsive Design
Even though there is a fair share of mixed-reality headsets out there, many fail to overlay digital elements on top of the real world properly. However, Apple’s headset has no such problems. Apple’s Vision Pro comes with a seamless 3D layered UI. This overlay effortlessly wraps on top of the natural world using advanced tech. However, the best thing about it being three dimensional is that the Vision Pro’s digital overlay is dynamic, which means it reacts to natural light and casts virtual shadows on surfaces.
This is made possible through a sophisticated LiDAR sensor that, along with front cameras, performs 3D mapping of your environment. This intricate level of 3D layering translates to a truly immersive experience where the user forgets the distinction between their world and the layered one. This responsive design is also fully user scaleable, and as such, can be easily carried over to general work, entertainment, and even gaming use.
Apple Vision Pro’s incredible immersive-ness through its sensors is quite unlike anything the MR market has ever seen. Provided the real-life hands-on live-up to the incredible demos, the all-new 3D design will be one of Apple Vision Pro’s best features.
2. Groundbreaking visionOS
The 3D layered responsive design we discussed above is one of the many interactable UI elements but features more. To provide a suitable platform for the headset, Apple has launched a dedicated operating system called visionOS. The new OS has been built from the ground up for spatial computing and includes a lot of features.
This includes many features like the three-dimensional interface, redesigned app store, and immersive environments. To tell you all about it, we have compiled a dedicated guide that’s linked above.
3. Displays Better than 4K TVs
As a Meta Quest 2 user, I’ve always had reservations about its relatively low-resolution display that causes eye strain and blooming around the edges. While the Vision Pro did fix that to an extent, it’s not fully there. However, the displays inside were one of the Apple Vision Pro’s best features.
The headset comes with dual micro-OLED panels with a combined total of 23 million pixels. For those unaware, that is a stunning level of pixel density. The displays also support a wide color gamut and high dynamic range. And while the exact resolution and brightness were not shared, the company has claimed that each display will have a “ better resolution than a 4K TV.”
This is a welcome addition to the MR headset, as without displays that can deliver the required visuals, the promised experience would be useless.
4. Environments Support
Even though the Apple Vision Pro is a mixed-reality headset, it naturally has the capability to turn your real world into a virtual one. While the headset has a UI overlaid over the real world, it can be traded in for some beautiful landscapes.
The Vision Pro comes with a feature that the company is calling “Environments.” Enabled through a twist of a dedicated digital crown, these environments are beautiful and immersive landscapes that will surround the headset wearer.
The intensity of the environment can be adjusted through the crown and either made minimum or entirely like a virtual world. The best part is that the foreground process doesn’t stop, no matter what you do. So, whether you are on an airplane watching a movie or busy in a meeting, you can swish the crown and watch the world behind you melt away.
5. Apple Eyesight
While it’s easy to be lost entirely in the Vision Pro’s environments, Apple has made sure that outside people gain as much focus as the ones inside. For example, the MR headset has a unique feature called “Eyesight” that shows the inside user’s eyes to the outside people. This is made possible through an external display that lights up, giving the impression that the Vision Pro has turned transparent.
Apple Eyesight intelligently turns on when it senses an outside person in proximity to the user. It even shows visual cues to the non-wearer about what the headset user is doing. The Vision Pro user, on the other hand, will see the outside user in full detail layered over their existing environment and UI.
Apple has stated its intention behind Eyesight is to ensure that the people around you stay connected with you even with the Vision Pro on. The forethought behind this makes Apple Eyesight one of the Vision Pro’s best features.
6. Spatial Audio
The multimedia experiences highlighted above will naturally be best experienced with audio that surrounds you completely. So while the Apple Vision Pro headset doesn’t have surround sound audio, it will still deliver the same experience. This is made possible by dedicated “dual-driver audio pods” inside the headset headband.
These audio pods use Apple’s spatial sound technology that virtually enhances the soundscape. This makes virtual surround possible and convinces users that the sound is coming from around them. However, it is worth noting that these audio pods are not in-ear and might not be the cup of tea for the audiophiles out there. Nonetheless, we expect these pods to be one of Apple Vision Pro’s best features.
7. Capture Spatial Photos
Even though the Apple Vision Pro has spatial audio, surround sound isn’t really anything new. However, hearing the words spatial photos surely drum up curiosity. This is one of the features that precisely sounds like one of the best ones of the Apple Vision Pro. Put simply, the Vision Pro uses its array of front-depth sensors to capture 3D like photos and videos.
This means that content captured by the Vision Pro will feature an added level of depth and realism, making them tangible in a sense. The captured photos and videos can be easily viewed anytime, and you will feel like you’re right there in that moment, according to Apple. The demo showcased the same as the Vision Pro wearer stood still while the headset mapped out the scene and captured it all.
It is currently unclear whether you can transfer spatial photos off the Vision Pro onto your iPhone or Mac. However, we imagine support for that should also arrive when the headset releases.
8. Dedicated Chip for Low Latency
Speaking of delivering the required visuals, Apple talked about the difficulty a headset can encounter with its need for so much power. As the Apple Vision Pro performs so many tasks simultaneously, it naturally needs a high amount of power while keeping things fast. For the first time, Apple uses a dual chipset combination of the M2 Silicon with a brand new R1 chip.
The trusty M2 chipset will drive computational power inside and keep the headset cool through its intake vents. However, the R1 chip is completely dedicated to keeping the visual latency at the bare minimum. Using the new in-house silicon, the Apple Vision Pro will have a mere latency of 12ms, which Apple claims is 8x faster than the human eye.
This will ensure that users see things exactly when they happen without lag. In addition, the R1 chip will automatically process input from the headset’s 12 cameras, 5 sensors, and 6 microphones to make this happen in real-time. So you can kiss any input lag goodbye.
9. Needs No Extra Equipment
One of Apple’s best features and boldest moves is that the Vision Pro will come with no controllers or extra equipment. Instead, the user will use a combination of the user’s hands, eyes, and voice to operate the headset entirely. The headset uses its built-in four depth sensors to intelligently read the user’s finger and wrist gestures to register an input.
Even though the Meta Quest Series comes with hand gestures of its own, it’s not good enough to entirely ditch the controllers. This is a bold move by Apple to fully rely on its combination of camera and sensor hardware and not to have controllers at all. Nonetheless, this is one of Apple Vision Pro’s best features, in our opinion.
10. Lifelike Realistic Digital Avatar
Even though Facebook has brought about digital avatars in Instagram and WhatsApp that mimic your facial expressions, they don’t hold a candle to what the Vision Pro showcased recently.
Using a combination of front sensors on the headset and advanced machine learning, the Pro can create an incredibly lifelike digital avatar that looks almost exactly like you. This avatar can then be used in Facetime to chat with Vision and non-headset users.
Apple claims that the neural network behind the process has been trained on “thousands of individuals.” From the showcased demo, the rendered avatar looks remarkably alike and dynamically matches the person’s facial and hand movements effortlessly. To match the expressions in real-time, the Apple Vision Pro also combines internal IR cameras and downward ones to make it possible. As a result, the created avatar has an extreme amount of depth and, in a way, reminds us of Project Starline from Google.
11. Unlock Vision Pro with Iris Scanner
The IR cameras inside the headset don’t just scan your face for Facetime calls. Apple has revolutionized privacy by bringing Optic ID, i.e. real-time eye scanning to the Vision Pro, making it one of its best features.
This new authentication method analyzes a user’s iris intricately. The headset then compares the iris to the enrolled data and decides if it matches. To keep privacy paramount, all the data is processed on the Vision Pro itself through a Secure Enclave Processor.
To add even more security and privacy to the headset, the sensors in the headset won’t share what you are looking at with third-party apps. It is only when a user selects something that the data is shared. Combined with Optic ID, this is truly an unprecedented move that brings a new level of privacy to MR headsets.
