Mar 25, 2016
Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries.
Problem: How do I measure graphics performance?
Frame rate is a good indicator of graphics performance, but this data is not always available. Some apps display frames per second (FPS) on-screen, but most do not. If your app does not have a frame counter, you’ll need to find software that does this. Unfortunately, getting accurate FPS readings is not easy – unless you know which app to use. There are quite a few apps on Google Play that claim to do this, but the most popular ones either require that you root your device, or connect your device to a computer that is running a special software service. Even then, the accuracy of the frame rates reported varies greatly.
Solution: App Tune-up Kit
Let’s start with a step-by-step guide on how to get accurate frame rates using App Tune-up Kit. First, make sure you have Snapdragon Profiler installed on your computer. You can download Snapdragon Profiler for Mac, Windows or Linux on the Qualcomm Developer Network. Snapdragon Profiler is required, because it can communicate directly with the graphics driver and obtain accurate frame rate data. You also need to make sure your computer is communicating with your Android device over ADB. If you haven’t done this already, there are articles like this one, which walk you through the process and show you where to download the proper drivers. Lastly, you need an Android device with a Snapdragon 805 (or later) processor running Android 5.0 (or later).
One you confirm that you meet all of the above requirements, here are the steps to get accurate frame rate readings on Android apps:
- Launch Snapdragon Profiler on your computer.
- Select ‘App Tune-up Kit’ from its Tools menu. A message will let you know when you have connected and can disconnect your mobile device from your computer. This is required in order to obtain accurate direct power readings. Click on OK to continue.
- Next, open App Tune-up Kit on your mobile device. If App Tune-up Kit was already running in the background, you’ll need to swipe it from the Recent Apps list, to make sure it is fully closed and start over at Step 1.
- Touch the Gear icon on the upper right to enter the Settings screen. Change the ‘GPU Profiling Mode’ setting from ‘GPU Load’ to ‘FPS (when available).’ Now you’re ready to view the frame rate of an app using App Tune-up Kit
- Select an app, which will automatically be launched, and use it like you normally would.
After 60 seconds you should see a dashboard appear like the one above. In most cases you should see an FPS value when you touch the GPU tab on the dashboard. In general, the higher the frame rate the better, because it indicates smoother motion, but there are times when software developers should limit frame rates to improve performance or reduce power usage. Refer to the Adreno Developers guide for guidelines on this.
If you see “N/A” under GPU, it means that the profiled app did not provide FPS data. App Tune-up Kit currently can only obtain framerate data from Open GL-based apps like games and multimedia apps. In other words, don’t expect to get FPS values from a calculator app. Here are some of the apps that we’ve confirmed show FPS readings in App Tune-up Kit: Basemark ES, BSPlayer, Chrome, Epic Citadel, GFXBench, Instagram, MX Player, Play Movies and TV, Subway Surfers, The Chase (Unity demo) and YouTube.
The best practices for this feature are consistent with those described in the App Tune-up Kit FAQ: close everything running in the background, use the app being tested like you normally would, wait until App Tune-up Kit finishes profiling before touching the Back button, and profile each app at least twice and retest if the average (or median) varies more than 10% between tests.
Testing 1, 2, 3…
Next we’ll compare the reported results from App Tune-up Kit with two other popular apps from Google Play that report FPS values to see how they compare with each other. To do accurate side-by-side frame rate comparisons you need a game loop or an app that is designed for graphics tests. For our first test, we’ll use a GPU benchmarking app called Adreno Test. It is ideal for this type of testing because there is no load-time and almost no variation between tests. Adreno Test also outputs accurate FPS data to logcat, so it can be filtered and exported to Excel. Once the data is in Excel, median, average and max FPS values can easily be calculated. For consistency, we’ll use a Nexus 6 running Android 6.0.1 on all of our tests. Here are differences between the three apps:
- App #1 - When I launched this app for the first time, it displayed a URL where I had to go to download a JAR file to be installed on my PC. After launching the software on my computer, I had to select my device, start the service. Finally I was ready to select the app I wanted to test and touch the ‘Start FPS Benchmark’ button on my mobile device. Adreno Test appeared on my mobile screen with a FPS meter in the upper right. App 1 showed FPS readings that bounced around between 53, 55, 57, 59 and 61 FPS. It didn’t display an average or median so I had to compute those values from the data I collected.
- App #2 - When I launched the second app from Google Play, it asked me to relaunch its desktop client by typing in a URL on my browser. This prompted me to download and launch a JAR file and to unplug my device. I then selected the Adreno Test app and let it run for the recommended time period. No on-screen FPS values were displayed by this app. After the test was completed this app displayed a median value of 38 FPS. By viewing its graph, I could see the max FPS was 55.
- App #3: App Tune-up Kit - The setup instructions for App Tune-up Kit were already covered above, but like the other apps, it requires an initial USB connection and the installation of special software on your computer. App Tune-up Kit doesn’t show real-time FPS values, so you have to wait until it finishes profiling to view the results, which are shown in an onscreen report. It reported an average FPS of 87.7, a median FPS of 93 and a max FPS of 100.
How Accurate Are The Results?
You can see the results reported by each app above. The error is the difference in percentage between the reference app (e.g. Adreno Test) and the app under test. In this test App Tune-up Kit is more accurate in every category. This is not necessarily because it’s a better app, it has an advantage that no other commercial apps have because it can access the graphics driver directly.
Many game developers use middleware from Epic or Unity so I also tested Epic’s Citadel app. It has a benchmark mode that provides an average FPS value when completed. This test is even less scientific than the first one since I had to use a video recording to get FPS data from Epic Citadel and App #1. The Epic Citadel average and max values are accurate, but median is less accurate because I had to compute it from a limited number of samples that I could read on-screen. Still, the results from both tests are consistent and show there are advantages when reading frame rate data directly from a graphics driver as opposed to getting it from dumpsys or SurfaceFlinger.
When it comes to reporting accurate FPS values, App Tune-up Kit comes out ahead in most tests. It’s also worth mentioning how the different apps handle maximum frame rates. Notice how the maximum frame rates of App #1 and App #2 never exceed 60 FPS – even though the reference apps reported maximum frame rates between 79 FPS and 97 FPS. This greatly limits their accuracy on this type of test with some apps. There are definitely times when it makes sense to use App #1 or App #2 instead of App Tune-up Kit. App Tune-up Kit also doesn’t currently report FPS readings on some types of apps. That is one area where the other two Google Play apps shine. The bottom line is that all of these apps have their place as test tools. The more you understand their strengths and limitations, the more you can get out of them. Happy frame rate testing!