Enabling augmented reality in unknown environments.

Qualcomm Research’s computer vision efforts are focused on developing novel technology to enable augmented reality (AR) experiences in unknown environments. A critical step in enabling such experiences involves tracking the camera pose with respect to the scene. In target-based AR, a known object in the scene is used to compute the camera pose in relation to it. The known object is most commonly a planar object, however it can also be a 3D object whose model of geometry and appearance is available to the AR application. Tracking the camera pose in unknown environments can be a challenge. In order to mitigate this challenge, We have developed a leading technology known as SLAM, which enables AR experiences on mobile devices in unknown environments.

Qualcomm Research has designed and demonstrated novel techniques for modeling an unknown scene in 3D and using the model to track the pose of the camera with respect to the scene. This technology is a keyframe-based SLAM solution that assists with building room-sized 3D models of a particular scene. The system runs a computationally intensive non-linear optimization called bundle adjustment to ensure that models with a high-level of accuracy are generated. This optimization is significantly improved using SIMD on NEON to make certain that a smooth operation occurs on mobile devices.

The system is designed to support a responsive AR application by delivering a camera pose in real-time at 30 Hz. In order to realize this technical challenge, the proprietary SLAM solution utilizes multiple cores in a multi-threaded architecture.

During fast motion tracking failures are typically experienced. In order to recover from such tracking failures, the system has a state-of-the-art relocalization routine which was developed to quickly compute an approximate pose from the camera when images are blurred or otherwise corrupted. The system also delivers a triangulated 3D mesh of the environment for the application to utilize in order to deliver a realistic augmented experience that is blended with the real scene.

A key factor in the development of this complex and responsive system, was for it to be very easy to use right from the start. As such, a unique initialization method for the SLAM system was developed which hides the user from the complexities of initializing a SLAM system. A camera pose is available to the application in the first frame, that enables applications to hide the initialization of the system from the user.