Enabling the metaverse with 5G

In my last blog post, I described how we’re in an exciting time to harness the low latency and high capacity of 5G to enable a new era of distributed computing —  where workloads can be split between the device, edge cloud, and cloud to enable the metaverse. We view the metaverse as a persistent spatial internet with personalized digital experiences that span the physical, digital, and virtual worlds. In this blog post, I’ll provide an update on the latest results from our boundless extended reality (XR) testbed, including our 5G modem application programming interfaces (APIs), and how 5G standards are evolving to enhance XR experiences.

Enhancing boundless XR experiences on our 5G testbed

Our boundless XR testbed utilizes industry-leading Snapdragon XR1, XR2, and XR2+ platforms, the Snapdragon Spaces SDK, and Snapdragon XR reference designs. It works across both private network and wide-area deployments, and across both sub-6 GHz and mmWave bands.

At MWC 2021, we demonstrated a multi-user 5G boundless virtual reality (VR) system using the Snapdragon XR2 platform with integrated 5G modem and the Ericsson 5G network. Boundless VR and mixed reality (MR) over 5G are now ready for deployment in private networks targeting enterprise, industrial, and consumer use cases.   

For the augmented reality (AR) device of the future, we want sleek, stylish, and light-weight glasses that provide immersive digital augmentation to enhance our lives. Distributed computing can help achieve this. At MWC 2022, we demonstrated how AR for wide-area networks can be enabled in the near term with Wi-Fi powered AR glasses paired with a 5G-enabled phone or companion device. 

The pose information and video flows from the AR glasses to the phone to the server. The server processes the data and sends back the encoded graphics via the phone to be displayed on the glasses. The round-trip time for this entire process is called motion-to-render-to-photon latency, or M2R2P. The 5G and Wi-Fi roundtrip times are key components of M2R2P. Our on-device optimizations help reduce the 5G and Wi-Fi round-trip times by more than 28 percent on average and more than 38 percent at the tail (i.e., 99th percentile), providing an improved and more uniform user experience. This is just the start — we are continuously innovating to enable new and improved AR experiences.

5G modem APIs for enhanced experiences

We have developed 5G modem APIs to enable applications to benefit from underlying modem optimization and low latency modes, leveraging features like packet prioritization, improved beam management, and dynamic packet processing across 5G protocol stack. In addition, our 5G modem APIs can provide real-time information about the network and radio frequency (RF) conditions, enabling applications to rapidly adapt bit rate and resolution to the changing environment. 

At MWC 2022, we demonstrated this API functionality in a real-time cloud gaming application that adapted video bit rate faster, improved video quality, and reduced stuttering in changing RF environments.

Figure 1: 5G modem API for improved application adaptation and user experience.

5G has a strong evolution path for enhanced XR experiences

5G has a strong roadmap that will further improve device power efficiency, device latency, and overall system capacity. For example, Release 15 introduced lower power modes like Bandwidth Part (BWP). Release 16 also provides enhancements like cross-slot scheduling, uplink configured grants, slot aggregation, and uplink skipping. Release 17 handles XR bursts better by allowing a faster transition to sleep and power savings by not monitoring the control channel. Release 17 reduced capability (RedCap or NR-Light) can also enable a category of low-power AR glasses. 

Figure 2: A strong 5G roadmap to further enhance XR experiences.

There are also several enhancements coming in 5G Advanced, starting with Release 18 proposals that can further improve the XR experience to offer lower latency, lower power, and higher capacity. For example, Release 18 allows 5G transmission times to be aligned to the multimedia cadence, such as 60, 90, or 120Hz, which was not possible in previous releases. This eliminates drift between DRX-ON and XR traffic, reducing latencies and device power consumption.

Figure 3: Release 18 proposals for improved XR experiences.

Release 18 also improves scheduler efficiencies by staggering device traffic arrivals from the application server to the gNodeB and by defining quality of service (QoS) based on multimedia payloads. Other Release-18 proposals for improving XR include retransmission-less configured grants, which enable the device to sleep after low-latency uplink transmission, and low latency mobility via L1/L2 signaling. Beyond Release 18, 5G Advanced will continue enhancing 5G boundless XR capabilities to bring more immersive experiences in the metaverse. Tune into my webinar to learn much more.

Path toward 6G

6G will further enable the merging of our physical, digital, and virtual worlds. Immersive interactions with XR will take human augmentation to the next level via ubiquitous, low-power joint communication and sensing. 6G XR requirements will be fueled by digital twins and spatial compute. Digital twins will make it possible to monitor, design, simulate, analyze, optimize, and predict the behavior of physical systems. The next level of spatial computing will be possible as perception, such as 6 degrees-of-freedom (6-DoF) head, hand, and eye tracking, advances to work in any environment regardless of lighting and mobility conditions.

Figure 4: 6G XR requirements are fueled by digital twins and spacial compute.

In addition, the 6G system design can natively support distributed computing as well as exploit synergies across perception, multimedia, and wireless. In our XR testbed, we have demonstrated that perception inputs, such as 6-DoF pose and depth maps, can be used to map RF reflectors and blockers, significantly improving mmWave beamforming accuracy.

This is just the beginning. 5G is a platform for innovation that is poised to introduce many new experiences. We’re excited to see our end-to-end 5G optimizations, new 5G modem APIs, and upcoming 5G features enable the mass deployment of low-latency experiences, like boundless XR, as well as other new and exciting use cases yet to be discovered.