Feb 16, 2018
Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries.
Here at Qualcomm Technologies, we distinguish ourselves from the competition by engineering leading, on-device AI technologies and inviting others to build on them. Our AI Developer Contest was no exception.
Last September, we asked developers around the world to submit concepts for an app which incorporated the on-device machine learning and artificial intelligence functionality found on the Qualcomm Snapdragon Mobile Platform. Winners of the first round were given a Qualcomm Snapdragon 835 Mobile Hardware Development Kit (HDK) to create an Android app based on their proposal.
One app stood out from the group of 20 finalists and was awarded the grand prize: A trip for two to Mobile World Congress 2018 in Barcelona, Spain or a cash payment of US$10,000. Created by Finnish developer Sergey Konvisarov, the Pencil Music App generates instrumental music in real time based on a user’s unique drawing. The integrated Qualcomm Snapdragon Neural Processing Engine (NPE) translates each line, squiggle, and shape to create a tune with drums, bass, and melodies. We spoke with Konvisarov about his inspiration for the Pencil Music App and his love for AI.
What compelled you to enter the AI Developer Contest?
I heard about the contest from CodeProject Daily Insider and learned there was a chance to win a Snapdragon 835 HDK for sharing a creative idea. I had already played with neural networks a bit, and I was intrigued by the Snapdragon NPE SDK for on-device AI.
What was the inspiration for the Pencil Music App?
We often play this game at home: My daughter draws something — a cat, dog, car, or person — and I try to guess the result and sound it might make. Then we swap. I thought it would be cool to have a robot watch us as we draw and generate crazy sound effects relevant to what we’re drawing. That’s when I came up with the idea of the Pencil Music App.
What was the development process like for your concept?
It was easy to come up with the idea, but it was hard to implement it. I first prioritized which features were important to have, then drew a simple architecture of module interconnections that involved the synth, composer, camera, AI, and draw pad. Then, I started coding.
To speed up the process, I worked on each module separately. When the deadline got closer, I married all of the parts together and started shaping a demo version.
At some point, I had to tell myself, “Stop. No more new features. Stabilize and finalize what you already have.”
What do you hope users gain from the Pencil Music App?
It’s an entertaining app for all music lovers, and hopefully, it’ll make people smile, calm them, or invigorate them. It could be educational for kids, and it might provide new ideas to musicians. I hope it will also be inspirational for other developers.
Do you have any plans to expand on the Pencil Music App?
I have an idea of fusing the core engine of the Airyware Tuner app with a deep neural network running on the Snapdragon NPE SDK. It will recognize the instrument being played and automatically associate the corresponding tuning profile, frequency range, and sets of analyzed harmonics with their AGC envelopes. Theoretically, it would help with instrument tuning when in a noisy environment. I also hope to employ a GAN/LSTM network for composing music, add more music styles, add ways to customize instruments, and add ways to share and export the audio clips.
How did Qualcomm Technologies’ solutions, specifically, help you develop your app?
I really enjoyed the ability to run the neural network on the GPU. The Qualcomm Technologies engineers did a great job enabling this possibility. And while my app did not require any external peripheries besides the camera and could be developed with a real phone powered by the Snapdragon 835, it would not be easy to debug and profile the app on a consumer device without rooting it first. With a dedicated Snapdragon 835 HDK, it was a real pleasure to debug and optimize the app’s performance.
What do you like about working with artificial intelligence?
I really like the fact that AI can do things that were previously impossible, such as translating languages in real time, synthesizing realistic images, composing music, and analyzing DNA. Recent achievements of AI have been awesome, and it is especially cool to be a part of the trend.
Congratulations to Sergey for winning the Qualcomm Technologies AI Developer Contest with the Pencil Music App. We look forward to seeing what he creates next.