The Microsoft HoloLens has finally received its first major update in over a year, which Microsoft calls the RedStone 4. The update provides a boost to developers using the UWP (Universal Windows Platform) to develop applications for the HoloLens with ease. Mixed reality has been a top emerging technology for at least 4 years now, but despite increasing investments in mixed reality, the current industry majors have yet to sufficiently commercialize a consumer product.
The headset which weighs around 580 grams, with no wires or oddly shaped antennas popping out of it, is surely on the heavier side of the spectrum of headsets, but the weight is evenly distributed and designed to provide great comfort. After the tragic failure of the much hyped Google Glass, the domain of HMDs (Head Mounted Displays) remains highly unpredictable as to what it carries for us to see in the future. The tech is new and is still in its nascent stage and with the launch of products like HoloLens and other HMDs, and it promises exponential growth in the near future.
The HoloLens, in contrast with most of the current technology on the market works on the concept of holographic imaging. The closest you can get to a HoloLens in terms of physical appearance is the now Facebook-owned Oculus Rift. The Rift aims at teleporting you to a whole new world where you see and interact with things and completely isolate yourself from the real world (virtual reality), whereas the HoloLens teleports things from the virtual world right in front of your eyes, for you to see it and interact with it. It won’t cause you to get isolated from your ‘real’ world, it just augments new objects to it, hence the name, Augmented Reality. Microsoft does not want to stick a screen to your face with this one, they want to stick a screen virtually everywhere you look, and create a Mixed Reality environment.
The possibilities, are of course, endless. From turning your entire living room into a Minecraft playground to getting real time instructions on how to fix the damaged solar panels on the International Space Station, everything can be made possible. It’s almost like Star Trek, the only difference being that only the person wearing the whopping $5000 headset can see the holograms, and everyone else just sees things as they are.
The tightly packed headset contains in itself a wide array of embedded sensors and the holographic display assembly. The depth sensing cameras analyze your surroundings and the combination of the CPU, GPU, and HPU (Holographic Processing Unit) use the holographic lenses to project images directly into your retina via the set of holographic lenses mounted in the front. The headset includes a built in battery pack, the capacity of which has been kept a secret by Microsoft, and it powers the entire headset for 2-3 hours, before needing a recharge. The entire set-up is completely wireless and uses Wi-Fi and Bluetooth for communication with other devices.
Microsoft revealed at HotChips, the annual semiconductor conference held at Stanford University, that the HPU chip used in the HoloLens is a 24-core DSP beast that uses Tensilica cores from Cadence. It is a custom made chip, designed specifically for the purpose and sports 8MB of SRAM and a GB of low power DDR3 RAM, all of this tech packed into a tiny 12mm x 12mm ball grid array package. The almost insignificantly sized package is said to perform calculations of the order of trillions every second. It achieves this marvel by using each of the 24 cores at same time to perform a different task by using multiprocessing. The CPU stacked in there next to the HPU is a 14nm Intel Atom Cherry Trail SoC, running a specially designed 32 bit version of Windows 10 and has its own 1 GB of RAM. All the current applications do not strain the HPU cores past 50% load, hence making the product future ready, yet the HPU leaves out only little work to be carried out by the CPU.
With the release of the HoloLens 2.0, Microsoft also announced their in-house AI co-processorwhich now comes built into the HoloLens, and learns about the user over time, thus continuously providing a better experience than the previous one, every time.
The HoloLens consists of a lot of small components packaged together to give the marvelous end product. But the heart of this product lies in the way it actually produces the holographic images. A technology called Liquid Crystal on Silicon (LCoS) enables Microsoft to achieve what they have achieved. Liquid crystal on silicon (LCOS) is a reflective microdisplay technology that is based on a silicon backplate. It is a combination of digital light processing (DLP) and liquid crystal display (LCD) projection technologies as it is reflective, but instead of using mirrors as in DLP, it uses liquid crystals that are applied on a reflective silicon backplate. Light is reflected off the backplate, while the liquid crystals open and close to modulate it.
The LCoS displays generate the light, which then passes through a set of imaging optics, and then through a waveguide, a combiner and a bunch of diffraction gratings which direct the light directly into your eyes. The waveguide is the most easily visible part of the assembly, being the transparent set of lenses mounted on the front. They carry the light from the light source to the position it is meant to be seen from. The diffraction grating, on the other hand is not a different component in itself. It is essentially just a coating on the surface of the waveguide that is used to direct light rays into the eyes.
Microsoft talks about this technology in its patent US9164290, which details on the invention titled ‘Grating configurations for a tiled waveguide display’, and explains how a set of 3 gratings can be used to create a field of view from the light travelling through a waveguide.
Patent US9091851, on the other hand talks about how light is guided from the light source, into the waveguide and then mixed with light from the surroundings to power the augmented reality based head mounted display assembly.
The HoloLens also uses Infrared sensors to track the movement of your eyes and adjusts the generation of images accordingly. It does so in order to keep the holograms in a fixed relative to the eye, failing which would cause the holograms to randomly fly around and end up making the user extremely nauseous. The HoloLens also has the capability of adjusting its interpupillary distance in order to fit a wide range of variation in face structures across demographics.
The HoloLens runs only on Windows 10 (for now) and also comes with Microsoft’s digital assistant Cortana which is capable of doing all the basic chores you’d expect out of a virtual assistant. A lot of applications and use cases have been teased by Microsoft, two of them being HoloGuide and HoloHear. The concept of HoloGuide being centered on guiding the user through an area with low visibility, which utilizes the spatial awareness feature of the HoloLens, whereas that of HoloHear being a real time translator for the deaf. It is supposed to translate in real time, from speech to sign language.
With most of the journalistic hype focusing on consumer applications, most notably gaming, Microsoft has been in the parallel attempting a different path – i.e. focusing on enterprise and industrial applications.
The medical industry, for example, is expected to gain a lot from the possibilities offered by HoloLens. From allowing young doctors and surgeons to seek training in a much more immersive and effective manner, Augmented Reality will also prove to be a great tool to increase the ease with which major surgeries can be performed, assisting the surgeons with a plethora of information right where it is needed. A clear view of the internal organs of the body may be achieved with minimal size of incisions.
The technology has also received a widely positive response from industries involved in space exploration and research. Astronomy has received a boost from the opportunities that HoloLens provides, by bringing the vastness of remote interstellar locations right at the fingertips of scientists. For example, researchers can get a clear 3D view from a cliff on Mars captured by a rover right here on earth, and record their observations with an increased level of accuracy, thanks to the added factor of spatial awareness.
Other anticipated applications may fall into the e-commerce industry, with virtual product demos allowing customers to get a full blown view of the products. This extends further into the automobile industry, where you could get a ‘closer look’ at the car you want to buy sitting in your living room, or even inside your own car.
The applicability of HoloLens to consumer arenas, including and mostly obviously gaming, is profound. More than just playing for a character, gamers can use HoloLens to experience authentic encounters in first person, be it shooting or racing. With Oculus Rift grossly underwhelming consumers (blame it on cost, lack of game titles or the inevitable nausea), it remains to be seen if Microsoft can solve those problems to make HoloLens the preferred instrument for gaming on its already popular Xbox platform. Notwithstanding however the gaming industry, HoloLens is uniquely positioned to change whole industries – which is still hard to imagine for other mixed reality players, including Oculus, Samsung, Sony and Leap.