Estimating Motion in Images

Image based motion estimation is a class of techniques which analyze sequential image frames and determine how each region of an image moves from one frame to the next. Being able to track object movement between image frames has a broad range of applications and use cases, including SpaceWarp-style frame extrapolation, used in gaming and spatial computing devices to improve the smoothness of the digital experience. Our Adreno Motion Engine support, powered by the Qualcomm Adreno GPU, has been previously covered here .

Aperture issue

One common challenge motion estimation can face is a phenomenon known as the aperture issue. As the tracking of motion relies on matching corresponding features between two images. If the content in a local region is dominated by strong or bold lines, the line feature tends to dominate the scoring of feature matches. This is problematic as the line is often nearly identical at any point along its length, causing it to be challenging for algorithms to determine exactly which point along the line is the true motion match and which points are simply very similar. The dominant nature of the line feature makes it difficult for fainter features and texture to provide a strong enough indication of the true motion to compete with the line content.

When this issue arises incorrect motion vectors can be produced, and when those incorrect motion vectors are used to perform frame extrapolation, there are jittery or jumpy artifacts where parts of the image along these lines moves in sporadic ways unrelated to the true motion in the scene.

Detect and Correct

Our Adreno Motion Engine has been improved to specifically target these aperture issues. During the process of performing motion estimation, we have interwoven algorithms which are able to detect these problematic areas and correct the erroneous motion. This interlocked solution eliminates the issue early, preventing the incorrect vectors from impacting the overall motion estimation during later stages of the pipeline and resolving the issue without needing applications or platforms to perform any additional post-processing passes over the data. The detection and correction is highly efficient, requiring only 5% additional cycles compared to the original solution.

Left: Note the jittery motions along the dark lines. Right: Extrapolated frames are stable and track the true motion using Qualcomm’s improved motion estimation.

Visualization of the improvements.

Conclusion

The enhancements made to Qualcomm’s Adreno Motion Engine bring a significant improvement in the quality of motion vectors and the visual quality of solutions such as frame extrapolation which are driven by them. Our customers in fields ranging from gaming to spatial computing and XR now benefit from the improvements, providing higher quality experiences to their users.

For a showcase example of the Adreno Motion Engine enabling high-quality frame extrapolation check out Virtual Desktop’s Synchronous Spacewarp feature, which includes these enhancements from v1.30.3 at https://vrdesktop.net

Interested in learning more about the Snapdragon XR2? Be sure to check out the Snapdragon XR2 HMD Reference Design.

For additional reading, check out our VR and Gaming Graphics blogs on Developer blog. You can also join our Developer Community on Discord to connect with fellow developers.