OnQ Blog

The new era of immersive experiences: making it possible

21. Aug. 2015

Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries.

We want immersive experiences. That’s the conclusion I’ve reached after asking people “what are the experiences worth having, remembering, and reliving?” I wasn’t surprised to hear things like: a live sporting event or concert, an exotic vacation, a great movie, or nature. These immersive experiences stimulate our senses—they draw us in, take us to another place, and keep us present in the moment. The question that I’ll answer in this blog post is: how technology is making our everyday experiences more immersive—whether we are playing a video game on our smartphone, video conferencing on our tablet, or watching sports on our virtual reality headset.

The broader dimensions of full immersion

Stimulating our senses is vital to immersion. The three key pillars that make an experience immersive are visual quality, sound quality, and intuitive interactions. While each pillar stands alone at making experiences more immersive—think of the importance of the visual quality when viewing a photo or sound quality when listening to music—they are also complementary and synergistic. Immersion happens when all three pillars are combined together. Each pillar has multiple dimensions that improve the overall quality. Full immersion can only be achieved by simultaneously focusing on the broader dimensions of each pillar.

Too often, the mobile industry has focused most of its attention on specific dimensions that do not necessarily provide the biggest bang for the buck. For example, visual quality isn’t only about the quantity of pixels, such as the resolution and frame rate. It’s also about the quality of pixels. Realistic color, sharpness, and just the right amount of brightness are equally critical to making experiences more immersive. Consider the importance of color accuracy for critical decisions, whether you are trying to pick the correct matching dress as a bridesmaid or the right paint for the living room.

For sound quality, the two key dimensions are high resolution audio and sound integrity. When the sound is realistic and matches the visual, you can be truly immersed in the experience. In contrast, when the sound quality is compromised, we immediately notice it—think of a movie where the audio was muffled or mismatched with the lip movements.

For intuitive interactions, natural user interfaces and contextual interactions are the two key dimensions. It is very hard to be immersed in a moment if you are constantly distracted or trying to figure out how to interact with your devices. Natural user interfaces, such as gesture and voice, allow us to seamlessly interact with devices. Contextual interactions allow devices to provide intelligent notifications and personalized experiences based on context. This keeps us in the moment and prevents unnecessary interruptions. For example, while you don’t want to be interrupted from a virtual reality gaming session because of a telemarketer’s phone call, you do want a notification when the pizza delivery person rings your doorbell.

Focusing on the right dimensions based on the device and user experience is the key to making experiences appropriately immersive. Getting this right is challenging since it requires an expertise in multiple technology vectors and meeting the performance, power, and thermal requirements. So what’s the right approach?

The optimal approach

The optimal way to enhance these broader dimensions of immersive experiences requires an end-to-end approach, heterogeneous computing, and utilizing cognitive technologies.

An end-to-end approach means thinking holistically at the system level, understanding all the challenges, and working with other companies in the ecosystem to develop comprehensive solutions. For example, the end-to-end approach is essential for maintaining color accuracy—one of the dimensions of visual quality—from camera to display. 

Heterogeneous computing uses specialized engines across the System-on-Chip (SoC) to meet processing requirements of immersive experiences at low power and thermals. For example, image processing tasks, like computational photography, use the majority of the processing engines in the SoC, such as the ISP, GPU, DSP, CPU, and display engine.

Cognitive technologies, like machine learning and computer vision, make experiences more immersive. For example, cognitive camera improves visual quality by automatically capturing better images and videos.

Qualcomm Technologies is taking this optimal approach and is uniquely positioned to enhance the broader dimensions of immersive experiences. Qualcomm Snapdragon processors are designed to provide an optimal heterogeneous computing solution by taking a system approach and custom designing specialized engines. For example, the new custom Qualcomm Adreno 530 GPU and Qualcomm Spectra camera ISP in Qualcomm Technologies' recently announced Snapdragon 820 are engineered to significantly enhance the visual processing capabilities to support next-generation immersive experiences related to computational photography, virtual reality, and photo-realistic graphics. Qualcomm Technologies also provides development and optimization tools to the ecosystem, enabling content creation and optimized devices. The goal is to see products with immersive experiences being commercialized as quickly as possible.

I’m very excited to see what’s possible in the new era of immersive experiences. Stay tuned for an upcoming blog post on virtual reality. I’ll explore whether it is really the ultimate level of immersion.

Want to learn more? Be sure to check out our Immersive Experience website and webinar. Also, sign up to receive the latest information about mobile computing in our Mobile Computing newsletter.

Qualcomm Snapdragon, Qualcomm Adreno, and Qualcomm Spectra are products of Qualcomm Technologies, Inc.

Pat Lawlor

Senior Manager, Technical Marketing

More articles from this author

About this author

Related News

OnQ

Heterogeneous Computing: An architecture and a technique

If you’re looking to create great mobile experiences, optimization isn’t optional: it’s a crucial step that helps transform good ideas into great execution. In our previous “Start Cooking with Heterogeneous Computing Tools on QDN” blog, we discussed the concept of heterogeneous computing and how it can help you get more from mobile hardware by sending computational tasks to the best suited processor. Heterogeneous computing is designed to help you achieve better application performance while improving thermal and power efficiency.

However, not all systems capable of heterogeneous computing are created equal and it’s important to understand why. Heterogeneous computing is both a computational technique and a hardware architecture. To achieve greater benefits, you are better served with hardware architected for heterogeneous computing from the ground up along with a software stack that facilitates heterogeneous computing techniques. It’s the combination of purpose-built hardware and a software stack offering granular control within a larger framework of system abstraction that allows for the deep optimizations that heterogeneous computing can deliver.

The Qualcomm Snapdragon Mobile Platform is designed on these principles. This starts with the microarchitecture – the choices made in platform circuitry that include how individual processors are engineered for high performance and how to optimize compute paths between the processors. Let’s look at the main components of the Snapdragon mobile platform and a few of the microarchitecture considerations that went into its design:

Qualcomm Kryo 280 CPU

Designed to handle complex workloads like web browsing and in-game artificial intelligence, the Kryo 280 features an octa-core processor with independent high efficiency and high performance core clusters. During normal operation, the high efficiency cores run most tasks while the high-performance cores activate for anything needing more power.

Qualcomm Hexagon 682 DSP

With the Hexagon wide Vector eXtensions (HVX), the Hexagon DSP excels at applications requiring heavy vector data processing, such as 6-DOF (or Degrees of Freedom) head motion tracking for virtual reality, image processing, and neural network computations. With a 1024-bit instruction word capability and dual execution of the control code processor and the computational code processor within the DSP, Hexagon can achieve breakthrough performance without draining system power.

Qualcomm Adreno 540 GPU

Ideal for arithmetic-heavy workloads that require substantial, parallel number crunching like 3D graphics rendering and camcorder image stabilization, the Adreno GPU is engineered to achieve improved power efficiency and 40% better performance than predecessors. Designed to deliver up to 25% faster graphics rendering and 60x more display colors compared to previous designs, the Adreno GPU supports real-life-quality visuals, and can perform stunning visual display feats like stitching together 4K 360 video in real time.

Heterogeneous computing in microarchitecture design

Beyond the performance enhancements among the individual processors, the Snapdragon mobile platform was designed to optimize the use of the processors together. For example, the Hexagon DSP can bypass DDR memory and the associated data shuffling CPU cycles by streaming data directly from sensors to the DSP cache. Similarly, the Adreno GPU supports 64-bit virtual addressing, allowing for shared virtual memory (SVM) and efficient co-processing with the Kryo CPU. These are just two of the microarchitecture design choices in the Snapdragon mobile platform that make it cutting-edge for heterogeneous computing.

Software

As we noted at the beginning of this post, heterogeneous computing is also a technique. And to truly support heterogeneous computing requires a software stack that provides developers the abstractions and the control to leverage the optimizations in the hardware per the requirements of their application.

To program the DSP or the GPU for heterogeneous computation, and to maximize their performance, developers can use the Qualcomm Hexagon SDK and the Qualcomm Adreno SDK, respectively. These SDKs open a toolbox of controls allowing for precision manipulation of data and computational resources.

For system-wide heterogeneous computing control, Qualcomm Symphony system manager SDK provides the software utilities designed to achieve better performance and lower power consumption from the Snapdragon mobile platform. Symphony is designed to manage the entire platform in different configurations so that the most efficient and effective combination of processors and specialized cores are chosen to get the job done as quickly as possible, with minimal power consumption.

On top of these SDKs it is possible for developers to build their applications directly – many developers opt for this route. However, there is a growing ecosystem of SDKs, frameworks and supporting libraries for accelerating development within a given application domain. Two examples of this are QDN's Adreno SDK for Vulkan for the Vulkan graphics API and our recently released Snapdragon VR SDK.

How to Put Heterogeneous Computing Techniques into Practice with Tools from Qualcomm Developer Network

Take advantage of the heterogeneous computing microarchitecture and software tools to get the most performance for your application using the Snapdragon mobile platform.

The Snapdragon mobile platform’s microarchitecture, software development kits and high-level frameworks layer upon each other to provide developers ultimate control of their application performance and power usage. This combination of hardware purposely designed for heterogeneous computing and software allowing the use of heterogeneous computing techniques is at the heart of delivering great user experiences.

Whether you’re a developer that has always written code that executes serially and are interested in how to push more work to other processors, or whether you already have heterogeneous computing know-how and want to use it more effectively – perhaps wanting precise control over the dynamic distribution of your workloads – visit QDN for the resources you need to level up your heterogeneous computing skills.

 

Qualcomm Snapdragon, Qualcomm Kryo, Qualcomm Hexagon and Qualcomm Adreno are products of Qualcomm Technologies, Inc.

 

23. Mrz. 2017