May 31, 2017
Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries.
We envision a future where technology will become more personalized and even more useful — one where your world will be intuitively augmented to fit your needs, and it will all happen seamlessly wherever you are. Over time, the technology itself will effectively disappear, as a new type of almost imperceptible mobile device emerges to replace nearly all others and change how we interact with the world. While there are many technology hurdles to overcome before all of that happens, some of it is already just starting, and maybe even sooner than you think through a convergence of technologies under Extended Reality (XR).
XR is an umbrella term encapsulating Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), and everything in between. Today AR and VR are distinct experiences, but they share many of the same underlying technologies that will power revolutionary XR experiences as AR and VR eventually converge into a new type of wearable device that’s capable of supporting both. XR will transform everyday consumer experiences and many market verticals, from industrial manufacturing and health care to education and retail. The possibilities are endless with XR. Mobile XR has the potential to become one of the world’s most ubiquitous and disruptive computing platforms of the next decade, just as the smartphone of this decade has become. And it’s why we are determined to accelerate the development of the key technologies required for XR adoption. It’ll only happen if technology leaders work together with the consumer in mind. Therefore, it is our hope that we can collectively cast aside the dogma of the past, save some breath, and make XR happen together.
The XR wearable will be a converged device
At some point in the future, we see the convergence of the smartphone, mobile VR headset, and AR glasses into a single XR wearable. In this scenario, a single pair of XR glasses will be used primarily for AR but will also be used occasionally for VR — the otherwise transparent glasses will have a mode where they become completely opaque and enter a “VR mode.” The XR glasses will of course need to be sleek, comfortable, and fashionable while providing immersive, cognitive, and always-connected experiences that truly enhance and enrich our lives. XR glasses could replace many other screens in your life — even big ones like the TV in your living room. We believe at some point in the next decade, they’ll eventually become the most commonly used device for conveying audio and visual information to humans worldwide.
Solving the key XR technology challenges ahead
All that said, XR will not happen without substantial technology innovation. There are many technical challenges that need to be solved before XR reaches mainstream adoption, but for now let’s just focus on these five:
- Display: XR needs a disruptive revolution in display technology. XR will require richer visual content and the ability to seamlessly switch between virtual and real worlds. Besides being comfortable and durable, XR displays will need to solve a myriad of challenges, such as the vergence-accommodation conflict. This is a physiological characteristic of the human visual system that if not properly accounted for, causes fatigue and discomfort when your eyes rotate and focus on objects at different perceived distances. It will also be a substantial challenge, albeit not an insurmountable one, to have the same displays enable application-controlled transparency and opaqueness, along with a wider field of view (FoV), substantial improvements in pixel density, higher frame rates, and increased brightness.
- Common illumination: Making virtual objects in augmented worlds visually indistinguishable from real objects is a tremendous challenge, but it’s a challenge that needs to be overcome for XR to reach its potential. We need further convergence between the disciplines of computer vision and 3D graphics to deliver common illumination between virtual and real objects. This way, for example, a virtual picture frame of your family that you choose to place on your desk (wherever you are sitting) will be indistinguishable from a real one, under any lighting condition.
- Motion tracking: Intelligent on-device tracking of our head, hands, and eyes is required to interact intuitively with our XR glasses and create immersion. For example, low-latency and precise inside-out 6 Degrees of Freedom (6 DoF) head tracking is required to meet the motion-to-photon latency of world-scale VR. Also, for most use cases, the best controller is no controller, so hand tracking in AR-mode and creating realistic virtual hands in VR-mode are essential. We also think that the use of eye tracking for automatic inter-pupillary distance (IPD) calibration, more efficient 3D graphics and video foveated rendering for enhanced visual quality, and new user interfaces designed for XR to create more natural intent-based interaction will be essential.
- Power and thermal: Accommodating the always-on, compute intensive workloads of XR within the power and thermal constraints of sleek XR glasses is very challenging. Improvements in heterogenous computing performance per watt, battery technology, materials science, software efficiency, and standardization are required across the ecosystem to meet our vision of ubiquitous XR.
- Connectivity: XR also requires ubiquitous and seamless connectivity to the internet and cloud services, initially served by Gigabit LTE and complemented with Wi-Fi. As we see more and more XR users with higher-quality immersive services, we will need the additional capacity that 5G provides. The streaming of XR video is expected to become the killer 5G use case in the next decade. As an example, XR video can require gigabits of bandwidth and very low latency for demanding 6 DoF video experiences. 5G enhanced mobile broadband will provide multi-gigabit throughput, as well as over-the-air latency down to 1 millisecond to help to improve the speed of interaction with streaming XR video applications. 5G will also help deliver a more uniform XR experience since data rates with 5G will not vary as much as they currently do today, when you are further away from the base station.
Accelerating XR adoption
This is a big challenge. We know that we can’t do this alone. Achieving our XR vision will take significant R&D and cooperation amongst the world’s top mobile hardware and software technology companies. However, Qualcomm has already started to work with some of these companies to embark down the XR path together, with a goal to collectively solve some of these difficult technical challenges that will eventually enable a future where XR glasses will progressively become sleeker and offer tremendously more functionality. We're currently in the early days of a multi-decade XR evolution — similar to the smartphone evolution from the bulky mobile phone to today’s sleek smartphone.
Here at Qualcomm, we’ll be doing our part to help drive XR forward. We are committed to delivering superior mobile XR processing at low power consumption. We’re developing the foundational technology required for XR, such as ultra-low power, immersive 3D graphics and camera subsystems with improvements specifically for XR, as well as converged computer vision, machine learning, intuitive security, Gigabit LTE, and 5G technologies to make future XR devices even more intuitive and connected. We’re also working with the ecosystem and investing in leading XR companies — all in an effort to make our vision of XR a reality sooner.