Overcoming the high power/high cost of computer vision.
Always-on context awareness is a primary imperative for smartphones, the Internet of Things (IoT) and future artificial intelligence applications. Just as vision comprises 83% of human perception, computer vision (CV) is an area that is attracting most of the technical attention and innovation. In IoT, however, where “low power” has emerged as the key slogan (who wants to replace the batteries for 8 billion IoT devices?), low-power CV implementation has been extremely challenging. A huge gulf exists between traditional CV implementations and the low power/low cost required for always-on IoT solutions.
Key Research Areas:
Revolutionizing always-on computer vision.
We have pioneered a Computer Vision Module (CVM) which can be integrated into a wide variety of battery-powered and line-powered devices, performing object detection, feature recognition, change/motion detection, and other applications. Our CVM contains a lens, a Complementary Metal–oxide Semiconductor (CMOS) image sensor, and a digital processor engine which performs 100% embedded processing within the module itself and outputs post-processed CV metadata (not an image) to a main processor via an SPI interface.
Operating at <2mW of end-to-end power, and expected to be sold at low cost, the CVM provides smartphones and IoT devices with affordable, always-on computer vision awareness. By emitting CV data about what’s happening in a field of view rather than transmitting images, the CVM also delivers a much more privacy sensitive vision solution.
The CVM ingests an image, interprets what’s happening in an image, and forwards the resulting metadata to the MP for decision making and action.
Engineering CVM breakthroughs: From smartphones to smart homes.
Face Detection for Smartphones
We are developing our CVM to enhance the user experience and convenience for devices and appliances. For smartphones, it provides passive, always-on face detection that can auto wake the device when a face is detected and auto sleep (save power) when a face is not detected. The CVM can also auto stop the screen brightness for as long as a face is detected. Additionally, the CVM also supports intelligent screen orientation where, upon face detection, it adjusts and holds the screen orientation based on the position of the user's face. The always-on CVM can also trigger third party applications and hardware. For example, it could enable a biometric-type iris authentication process to be initiated when a face is detected or a QR scanner could be initiated when a QR code is detected.
Beyond face detection, simple gestures can wake and trigger the smartphone handset. Lastly, the CVM can provide ambient light sensing (ALS) and proximity functionality.
Interactivity Trigger for Toys and Smart Appliances
We are designing the CVM as an Interactivity Trigger to be embedded in devices such as connected cameras, smart assistants, smart appliances, and toys to intelligently react to users. For example, it could enable a smart refrigerator door to turn transparent and light up when it detects that a user is interested in seeing what's inside, or it can enable an appliance screen to present a menu of options when it sees a user is interested in engaging with it.
Occupancy Trigger (OT) for Smart Home Devices
Our CVM's OT enables devices to much more accurately distinguish humans in a field of view. With a smaller form factor than passive infrared (PIR) sensors, and at a very low cost and power, the CVM will more accurately identify humans in a field of view, and track their specific motion, noting whether they are moving toward the device or perpendicular to the device. It also discriminates between them and other moving objects, such as vehicles and pets, which normally trigger PIR sensors. The OT also provides associated metadata about the humans, reporting on variables such as location and the number of people within the sensor's field of view. Lastly, the highly flexible OT is trainable for varied types of object detection such as vehicles or pets using machine learning approaches.
Standalone Data Tracker (SDT)
Our CVM is being engineered with a coin cell battery and Bluetooth® radio into a stamp-sized SDT that can be used in commercial, residential, and smart city applications.
Our CVM’s extraordinarily small form factor allows it to be inserted into countless types of devices supporting numerous applications.
If you find the work we’re doing in computer vision to be exciting, and you have a technical background in CV, machine learning, low-powered sensors, or digital processing, we’d love to hear from you. Please visit us at www.qualcomm.com/company/careers to submit your resume. When creating your Qualcomm profile, please enter the activity code, "CV".
To learn more about our Always-On Computer Vision Module, please contact CVM@qti.qualcomm.com.