UL Delists MediaTek-Powered Phones with Suspect Benchmark Scores
April 15, 2020
No one benefits when a company tries to manipulate benchmark scores to its advantage. That's why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software.
Unfortunately, benchmark boosting continues to be a problem in the smartphone industry, despite the past scandals when manufacturers have been caught.
Last week, AnandTech reported that several smartphone models powered by MediaTek processors were producing artificially high and misleading benchmark scores. MediaTek is one of the largest fabless semiconductor companies in the world.
After reviewing AnandTech's benchmark data, we have delisted over 50 smartphone models powered by MediaTek processors. Delisted devices appear unranked, and without PCMark for Android scores, at the bottom of our popular list of the best smartphones.
Benchmark scores from delisted devices should not be used to compare models.
Which models have been delisted?
Based on AnandTech's data, we have temporarily delisted the PCMark for Android scores for all devices that use the following MediaTek chipsets:
- MediaTek Helio G90
- MediaTek Helio G70
- MediaTek Helio P95
- MediaTek Helio P90
- MediaTek Helio P65
- MediaTek Helio P60
- MediaTek Helio P20
- MediaTek Helio A22
In total, over 50 devices from more than 25 different vendors have been delisted including several recent, high-performing models such as the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.
Why has UL delisted devices with these chipsets?
AnandTech tested the Oppo Reno3 Pro with MediaTek Helio P95 chipset with the public version of PCMark for Android, available from Google Play, and a private version of the app that is not available to the public or manufacturers.
AnandTech found a big difference in the scores from the public and private apps, even though the tests in each are identical. In AnandTech's words,
"...the two versions of the benchmark did differ in their scores – but I was still aghast at the magnitude of the score delta: a 30% difference in the overall score, with up to a 75% difference in important subtests such as the writing workload."
With a bit of digging, AnandTech found a benchmark whitelist in the device's firmware files. Further investigation found similar whitelist files on a host of devices powered by MediaTek chipsets,
"Inspect the file, there we find amongst what seems to be a list of popular applications with various power management tweaks applied to them, with lo and behold, also a list of various benchmarks. We find the APK ID for PCMark, and we see that there’s some power management hints being configured for it, one common one being called a 'Sports Mode.'"
We recommend reading the full article on AnandTech, but in summary, the device uses a hidden "Sports Mode" and other settings to artificially boost its performance while running the public PCMark for Android app. With the private app, the phone performs as expected for normal, everyday use.
The difference in scores tells us that the device is simply recognizing the PCMark app by name rather than adapting to the type of work in the test. This kind of detection and optimization is forbidden by our rules for manufacturers.
Optional performance modes that can be set by the user—already available on some other manufacturers' models—are allowed under our current rules as long as they are disabled by default. A device must run the benchmark as if it were any other application.
Benchmark rules and industry standards
We contacted MediaTek with our findings, but did not receive a response. Shortly after AnandTech published its article, however, MediaTek responded with a post on its website.
While the response is mostly a point of view, we would specifically call out the following misleading claim,
"MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets."
Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an "accepted industry standard." It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device's true performance in everyday use.
Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.
As it has with similar cases in the past, we hope this delisting will help persuade MediaTek to change its approach and join the rest of the industry in adopting benchmarking best practices.
We're committed to creating benchmarks you can trust
- UL benchmarks are protected by rules for manufacturers.
- Our benchmarks have public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
Test your phone with 3DMark and PCMark for Android
3DMark is a free Android benchmark app for testing the gaming performance of your phone. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.
PCMark for Android is a free Android benchmark app that measures phone performance and battery life with tests based on common, everyday tasks such as browsing the web, editing photos and videos, and working with documents and data.
3DMark Steel Nomad will be free for all 3DMark users.
December 20, 2023
Using 3DMark to measure sustained performance.
December 13, 2023
New NPUs supported by Procyon AI Benchmark
November 16, 2023
A preview of 3DMark Steel Nomad
November 10, 2023
Qualcomm Uses Procyon AI Benchmark at Snapdragon Summit
November 1, 2023
3DMark celebrates its 25th year!
October 26, 2023
3DMark Adds iOS Support to Solar Bay Ray Tracing Benchmark
September 11, 2023
UL Solutions Launches 3DMark Solar Bay, New Cross-Platform Ray Tracing Benchmark
August 14, 2023
3DMark is now also available on Epic Games Store
June 20, 2023
Procyon® Is Addressing the Demand for Better AI Capabilities with Newly Launched AI Benchmark
June 15, 2023
Lenovo Focuses on Optimizing PC Hardware Testing with Procyon® Benchmarks
May 9, 2023
New Office Productivity Battery Life Scenario Released Today From Procyon®
May 4, 2023
Procyon® from UL Solutions Launch MacOS Benchmarks for Multi-Platform Score Comparison
May 2, 2023
3DMark adds feature test for AMD FSR.
March 30, 2023
UL Procyon® AI Inference Benchmark for Windows Released 22nd March 2023
March 20, 2023