The full saga of Apple’s troubled mixed reality headset has been revealed

A series of reports in The Information paint a detailed picture of politics and problems facing Apple's plan to develop a virtual, augmented, or mixed reality headset since the initiative picked up steam back in 2015.

Citing several people familiar with the product, including some who worked on it directly, the reports describe a contest of wills over the direction of the device. The standoff was between Apple's mixed reality product team (called the "Technology Development Group") and famed Apple designer Jony Ive and his industrial design team. The report sheds light on Apple's plans for the device, which Bloomberg recently reported is nearing launch.

They also claim that Apple CEO Tim Cook has been relatively hands-off from the product compared to others like the iPhone, and that the Technology Development Group's location in a separate office from the main Apple headquarters has been a source of problems and frustration.

The Information's sources say that Apple's mixed reality efforts began almost accidentally when the company purchased a German AR startup called Metaio to use some of its technology on Project Titan, its self-driving car project. Another key moment was when Apple hired the AR/VR project team leader, Mike Rockwell, away from Dolby Laboratories. Starting in 2015, Rockwell built a team that included Metaio co-founder Peter Meier and Apple Watch manager, Fletcher Rothkopf.

In 2016, several AR demos were shown to members of Apple's board. In one, a tiny triceratops grew to lifesize before the board members' eyes. In another, a room was transformed into an immersive, leafy environment. But the board was not Rockwell and the company's most significant barrier. According to The Information, it was Ive, who oversaw both the industrial design and human interface teams at Apple.

Ive and his crew argued against a VR headset because they believe VR separates users from the people and world around them, and that VR headsets look unfashionable. But the Technology Development Group gained the industrial design team's support by presenting a concept: an outward-facing screen on the front of the headset that showed images of the wearer's facial expressions and eyes to people around them. The wearer could see the people around them through an external camera feed.

Rockwell and his colleagues developed and released ARKit in 2017, an application development suite that allowed developers to make AR apps for iPhone and iPad using technologies and techniques that could be adapted to a headset later.

At first, Rockwell and the rest of the mixed reality team wanted the headset to be tethered to a base station to provide maximally impressive graphics and performance, and some on the team envisioned it as primarily a tool for professionals and creatives to use at their desks. But Ive didn't like either of those ideas and wanted it to be a mass-market lifestyle product that consumers could take on the go. Apple's most senior leadership backed Ive's plan, and Ive still has an active role in the headset's development, even though he now works with Apple as a consultant.

The decision to make the headset a standalone device reportedly caused significant headaches. For example, some felt the most optimal way to make it work well on its own would have been to put more functions on a single chip. But since the silicon work was already done, they had to find ways to battle the latency inherent in having multiple chips in the device communicate with each other. They had also been making software assuming that the base station plan would go ahead.

Nonetheless, the device is moving into the final phases of development. Bloomberg claimed last week that an advanced version of the product was recently shown to Apple's board, and that Apple has "ramped up" development of the headset's software, which is an iOS offshoot called rOS. (The R stands for "reality.")

The Information's reporting reveals many details about the upcoming headset. It would feature a resolution of at least 4K for each eye, which the team believed was the bare minimum for users not to perceive the image as pixelated, in contrast to most current consumer VR headsets. Its built-in processor would be closely related to the M2 processor expected to reach Macs and iPads in the coming months.

The headset would also have 14 cameras, some facing outward and some facing inward. It would allow users to see the outside world and enable nearby people to see a video representation of the user's eyes. It would live track the wearer's face and body movements to map to a 3D avatar (likely similar to the iPhone's Memoji) that could be used to have remote meetings and social gatherings with other headset wearers who are far away.

Because of the limited processing capabilities of the M2 chip in a headset without a tethered base station (the canceled base station was said to have ultra-high-performance M1 Ultra), the avatars would be cartoony. The Information's sources also say that more photo-realistic avatars were attempted when the base station was a part of the plan, but the uncanny valley was an issue.

Apple initially planned to launch the headset in 2019, but it now looks like it could instead be announced either later this year or in 2023. Additionally, Apple plans to introduce more natural-looking AR glasses as a follow-up product, but that device may still be years away from shipping.