The end of video buffering: Are we there yet?

Versatile Video Coding (VVC) is the most recent international video coding standard jointly developed by ITU-T and ISO/IEC. The VVC codec allows video data size reductions of around 50% for the same video quality compared to its predecessor, High Efficiency Video Coding (HEVC). Video compression is increasingly important in an era where network video traffic keeps increasing. With the upcoming tidal wave of generative artificial intelligence (AI) video content, as well as the likely increasing pace of video content production and consumption, reducing the number of bits to represent media remains a relevant problem.

Let’s look at some of the foundational innovations that set VVC apart from currently deployed video codecs and Qualcomm Technology Inc.’s actions in the pursuit of a vibrant VVC ecosystem.

Optimizing video coding for common user experiences

Beyond significant data size reduction, the VVC codec also introduces intelligent tools that are particularly helpful with certain video scenarios, such as video conferencing, screen sharing and omnidirectional video.

Video conferencing

The COVID-19 pandemic has had a profound impact on how we work, leading to a dramatic increase in the demand for remote working tools, especially video conferencing software. Video calling has become a very important form of communication for both personal and professional usage.

During a video call, network bandwidth on both ends is continuously monitored so inbound and outbound video quality can be adjusted to fit the video data into available bandwidth. With previous video codecs, such as High Efficiency Video Coding (HEVC) and Advanced Video Coding (AVC), when video resolution changes to adapt to available network bandwidth, the server needs to send a special type of frame, known as the Instantaneous Decoder Refresh (IDR) frame. This breaks temporal prediction, which means that data estimation from preceding and succeeding frames cannot be referenced.

With VVC, there is a special feature called Reference Picture Resampling (RPR) that remedies this shortcoming from current-generation video codecs. RPR allows changing the resolution of a video during decoding without needing an IDR frame. This can avoid data spikes caused by large IDR frames, leading to a smoother video calling experience without stalls.

Screen sharing

Screen sharing has become increasingly popular for collaboration between teams, especially in the age of remote work. With the ability to share screens, team members can easily present their work and collaborate on projects in real time. As the use of screen sharing has grown, so too has the use of graphics data that is rendered, mixed or overlaid together with camera-generated content. 

From its inception, VVC is a codec designed to support coding screen-shared content. The VVC codec includes five main low-level tools related to screen content coding: Transform Skip Residual Coding (TSRC), Block-based Differential Pulse-Code Modulation (BDPCM), Intra Block Copy (IBC), Adaptive Color Transform (ACT), and palette mode. 

These tools help improve the color accuracy and overall data efficiency of coding screen content by reducing the amount of data that needs to be transmitted. While previous video codecs added some screen-coding improvements, these were not as efficient and were not supported on all devices. In contrast, coding tools for screen-shared content defined with VVC are supported in the main profiles by default, which should be available in most VVC-enabled devices. 

It can be argued that VVC will be the main choice for applications having fully or partially screen-shared content, given the efficiencies compared to other video codecs.

Omnidirectional video

Omnidirectional video has gone beyond the passive paradigm of traditional video, offering higher degrees of immersion and interaction. This revolutionary technology enables users to interact with a scene or environment, and to feel a sense of engagement and presence in a virtual space. Virtual reality (VR) is an example of immersive technology that has already landed in our everyday life, with an impact across major economic sectors beyond entertainment, such as e-healthcare, e-education and cultural heritage.

VVC introduces intelligent tools for omnidirectional video: bitstream extraction and merging, sub-pictures, virtual boundaries, and horizontal wrap around motion compensation. Omnidirectional video typically requires simultaneous handling of several elementary streams in a time-synchronized fashion. With previous video codecs, each elementary stream would require the instantiation of a video decoder instance. However, bitstream extraction and merging defines a way for stitching and cropping videos in the compressed domain, which decouples the number of elementary streams required by the video application and the number of video decoder instances needed to run on the device. 

Sub-picture is a new picture partitioning scheme introduced in VVC. It is a coded rectangular region of a picture that can be extractable since it is coded independently from neighboring regions. A sub-picture can correspond to an area of interest, such as the region a consumer is looking at with a VR headset. This allows prioritizing and sending higher fidelity sub-pictures to a headset device based on the viewing direction.

In 360-degree video, there may be discontinuities at the unaligned face boundaries of some projections, e.g., cubemaps, when the 360-degree video is converted to a 2D representation. The ability to disable in-loop filters across the edges helps to reduce the artifacts. With VVC, in-loop filter control is done by horizontal or vertical virtual boundaries where the configuration of virtual boundaries is signaled in a bitstream.

Wrap-around motion compensation allows for motion compensation to be performed from outside of the left boundary of the picture, where prediction wraps around in horizontal direction and uses pixel values from the right side of the picture, and vice-versa. This new flexibility helps with coding using equirectangular projection of omnidirectional video. 

Leading foundational video technologies

For over 20 years, Qualcomm Technologies has played a critical role in the development of important video technologies that have revolutionized how visual content is captured, transmitted and consumed across smartphones, tablets, PCs, TVs and other devices. These innovations have resulted in enhanced video quality and richer content at reduced bandwidth and storage, creating enormous benefits for both consumers and companies in the ecosystem. Our inventors are widely recognized, and our inventions have impacted the world far beyond the mobile ecosystem. The video technologies we helped drive are widely deployed and include leading video codecs, streaming protocols, and multimedia file formats. 

VVC is no exception. Same as for HEVC, Qualcomm Technologies is the leading contributor to the VVC standards that we described in this blog.

Pushing the ecosystem forward with a smartphone VVC demonstration

VVC is gaining a lot of momentum in the ecosystem. It has already been added to the DVB core specifications for video coding in broadcast and broadband applications. ISDB-T International, or SBTVD, which is a technical standard for digital television broadcast in Brazil, has also adopted VVC as the main video codec — with other broadcast standards expected to follow.  

At Qualcomm Technologies, we are excited about how the VVC codec will uplift video experiences of today. To that end, we are already collaborating with the ecosystem to bring VVC technology to consumers. 

Most recently, we worked with Tencent to optimize their VVC software decoder on a Snapdragon 8 Gen 2 mobile processor utilizing the Qualcomm Adreno GPU. This pre-commercial prototype achieved performance gains of over 30%, enabling stable and real-time smooth playback of 4K 10-bit 60 frame per second (FPS) ultra-high definition VVC content. The first demonstration showcasing this optimized decoder will be at ChinaJoy 2023, held in Shanghai from July 28-31. We believe optimized software decoders, such as this decoder from Tencent, will help accelerate adoption of VVC codec by video content services. 

We will continue close collaborations with the ecosystem to help the proliferation of VVC technology. This will also include bringing an even further optimized VVC playback functionality to devices powered by Snapdragon processors, with hardware VVC playback support scheduled for future Snapdragon releases.