UL delists Oppo phones with suspect benchmark scores
October 18, 2018
After Huawei was caught unfairly boosting benchmark scores last month, we were contacted by reviewers from Tech2 who had similar concerns about Oppo smartphones.
Extensive testing by Tech2 found that the flagship Oppo Find X smartphone produced artificially high and misleading benchmark scores.
UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software. The purpose of these rules is to ensure that users get benchmark results that accurately reflect the true performance of the device for apps and games.
After reviewing Tech2's benchmark results, and conducting testing in our own lab, we have decided to delist the affected models and remove them from our performance rankings.
Which models have been delisted?
Based on Tech2's testing data and reporting, we have delisted the Oppo Find X. Based on our own testing, we have delisted the Oppo F7.
The Oppo Find X was ranked #4 in our list of the Best Smartphones for 3DMark Sling Shot Extreme performance. It now appears unranked, and without a score, at the bottom of our rankings. 3DMark scores from delisted devices should not be used to compare models.
Why have you delisted these models?
Each model was tested with the public version of 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers.
We found that the scores from the public 3DMark app were up to 41% higher than the scores from the private app, even though the tests are identical.
The difference in scores tells us that the devices are simply recognizing the 3DMark app by name rather than adapting to the type of work in the test.
This kind of detection and optimization is forbidden by our rules for manufacturers.
We contacted Oppo with our findings. Oppo admits that the devices are detecting the public version of the app,
"When we detect that the user is running applications like games or 3D Benchmarks that require high performance, we allow the SoC to run at full speed for the smoothest experience.
Oppo explained how its devices manage power and performance for other apps that are not recognised,
"For unknown applications, the system will adopt the default power optimization strategy...After the user has not actively operated for 5 to 10 seconds, the device limits the system performance to 70% to 80% of the maximum performance (according to different platforms)...When there is a user operation, it will immediately cancel the performance limit, to ensure that the user experience is not affected."
In practice, this means that it is possible to improve performance in most apps by continually tapping on the screen. However, this has no effect on the public 3DMark app. Nor do we consider such tapping an accurate way to benchmark a device due to the variability in performance that it introduces.
Fortunately, Oppo also said that it is looking to improve on this approach for the benefit of its users.
"At the same time, we are working on upgrading the system, and strive to distinguish between the requirements of undetected apps or the subjective needs of users."
Under our rules, optimizing performance by detecting heavy computation demands is allowed. Simply detecting the benchmark app by name is not. A device must run the benchmark as if it were any other application.
We're committed to creating benchmarks you can trust
- 3DMark is protected by rules for manufacturers.
- 3DMark has public documentation that explains what each test measures and how the scores are calculated.
- Our Best Smartphones list only shows publicly available models. We never list or leak unverified scores from pre-release hardware.
Test your phone with 3DMark
3DMark is a free Android benchmark app available from Google Play. Benchmark your smartphone then compare its performance with the latest models. With its unique charts, lists, and rankings, 3DMark gives you unrivaled insights into the performance of your device.
3DMark Steel Nomad will be free for all 3DMark users.
December 20, 2023
Using 3DMark to measure sustained performance.
December 13, 2023
New NPUs supported by Procyon AI Benchmark
November 16, 2023
A preview of 3DMark Steel Nomad
November 10, 2023
Qualcomm Uses Procyon AI Benchmark at Snapdragon Summit
November 1, 2023
3DMark celebrates its 25th year!
October 26, 2023
3DMark Adds iOS Support to Solar Bay Ray Tracing Benchmark
September 11, 2023
UL Solutions Launches 3DMark Solar Bay, New Cross-Platform Ray Tracing Benchmark
August 14, 2023
3DMark is now also available on Epic Games Store
June 20, 2023
Procyon® Is Addressing the Demand for Better AI Capabilities with Newly Launched AI Benchmark
June 15, 2023
Lenovo Focuses on Optimizing PC Hardware Testing with Procyon® Benchmarks
May 9, 2023
New Office Productivity Battery Life Scenario Released Today From Procyon®
May 4, 2023
Procyon® from UL Solutions Launch MacOS Benchmarks for Multi-Platform Score Comparison
May 2, 2023
3DMark adds feature test for AMD FSR.
March 30, 2023
UL Procyon® AI Inference Benchmark for Windows Released 22nd March 2023
March 20, 2023