OnQ Blog

Developing expansive video experiences on mobile

Thanks to today’s advanced mobile devices, video content is everywhere and being consumed in more ways than ever including high resolution 4K video capture/streaming and playback, interactive videos, XR environments (including virtual reality and augmented reality), cloud gaming and facial recognition to name a few.

Supporting these demands are features such as high density displays, onboard cameras, a myriad of ways to interact, and powerful edge computing capabilities. For developers this creates additional opportunities with accompanying challenges to provide your users with an optimized viewing experience. In this blog, we’ll look at how a high-end mobile platform can provide a solid baseline for delivering a wealth of mobile video experiences.

It’s all about the power

Despite the demand for ultra-sharp images and other cutting-edge features, mobile users continue to cite long battery life as one of their top requirements. So, when developers are starting a new project, power efficiency is something to consider. When evaluating a development platform, it’s important to identify how power can be controlled for specific types of tasks, both across cores and across processors at the platform level.

For example, the Qualcomm Kryo 385 CPU found in the Qualcomm Snapdragon 845 Mobile Platform is an ARM-based design with a big.LITTLE architecture allowing tasks to be dynamically allocated between powerful cores and energy efficient cores. We covered big.LITTLE in a previous blog, which you can review to see how it can impact your development. As with any platform, you should identify how much and what type of control you have over such allocation. In the case of Snapdragon 845, you can control the allocation directly if you know the performance requirements of specific tasks, or let the OS take care of things.

At the platform level, Snapdragon 845 was built around a heterogeneous design. By incorporating different processors like an ISP, DSP, GPU, and CPU, developers have a rich set of specialized processors, across which specific tasks can be allocated for the best possible energy efficiency and performance. To fully optimize this allocation, we recommend using our Qualcomm Snapdragon Profiler early on in your project so that you can monitor and evaluate key metrics such as CPU Core Load, CPU Core Utilization, power, etc.

Rendering also requires a lot of power, so look for methods and features that reduce both rendering times and power consumption. For example, with eye-tracking applications, the Qualcomm Adreno Foveation can intelligently render different parts of the screen at varying levels of detail relative to the fovea, designed to help to reduce the power required for rendering.

Storing hi-fi graphics

Today’s devices can capture, store and render imagery at unprecedented resolutions and clarity, which requires massive amounts of storage and memory.

For example, Snapdragon 845 supports an UltraHD Premium Rec. 2020 color gamut and 10-bit HDR for displays and cameras, and video capture of rates up to 240fps at 1080p or 480fps at 720p. With specifications like that, we recommend you start your project with some basic storage and bandwidth calculations.

Begin by first selecting a codec. Then, identify the codec settings for an acceptable level of detail loss due lossy compression, and calculate the average size of a compressed frame of video. This is a great article about codecs if you want a refresher. Then, determine the framerate you plan to support, factoring in the size of each compressed frame that must be transferred or stored, to come up with approximate bandwidth and storage requirements.

Edge cases

Edge computing is another area that is relevant to video-related experiences, so determine up front whether it makes sense to do the heavy computing on the device, in the cloud or across both. In many mobile applications, data is often processed as close to the source as possible (i.e., on the device) to reduce bandwidth consumption or the dependency on connectivity to access remote processes.

For example, facial recognition is a common use case in which machine learning and Artificial Intelligence (AI) are performed directly on a mobile device to identify faces from videos or pictures. For this type of edge computing, you’ll want a platform that can process imagery without burdening the CPU. On top of this, machine learning and AI are much easier to implement using proven frameworks like TensorFlow or Caffe2, so look for platforms that support these as well.

When you start developing for Snapdragon 845, you can use its Qualcomm Spectra 280 ISP to handle the image processing and its Qualcomm Hexagon 685 DSP for machine learning. And through Snapdragon 845 support for TensorFlow and Caffe2, you have a strong foundation for building intelligent video and image related applications.

Time to create your next video experience

As we’ve seen, today’s mobile video applications can provide a wide range of experiences, each with their own unique development challenges and platform requirements. Given the growing demand for mobile video, Qualcomm Developer Network is here to provide you with the tools and resources for optimizing your development on high-end mobile platforms to deliver the expansive video experiences that mobile users are now demanding.

 

Qualcomm Adreno, Qualcomm Hexagon DSP, Qualcomm Kryo CPU, Qualcomm Snapdragon, Qualcomm Spectra ISP are products of Qualcomm Technologies, Inc. and/or its subsidiaries.

 

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries ("Qualcomm"). Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.