The Samsung Galaxy S9 and S9+ Review: Exynos and Snapdragon at 960fps

System Performance

Raw CPU performance doesn’t always translate into actually better performance in-real world applications. Synthetic benchmarks are constant and long-running in their nature so performance response isn’t something that’s being tested. Real applications are a lot more bursty in their nature and might not only require high performance, but require high performance as fast as possible. Here a SoC’s scheduler and DVFS settings can have large impact on perceived “responsiveness”.

On Android, Futuremark’s PCMark is currently the best tool we have at hand to measure reproducible UI workloads. PCMark on Android makes usage of OS APIs and thus should be representative of workloads commonly found in applications.

Starting off with the Web Browsing 2.0 benchmark, the Snapdragon Galaxy S9 performs very similar to the QRD845 which we previewed in February. The performance boost for the Snapdragon 845 is 12% compared to the Pixel 2 which was the best performing Snapdragon 835 platform.

We’re off for a troubling start for the Exynos 9810 as it posts some disappointing scores in the web test. We first covered the surprisingly bad performance of the new SoC in our MWC on-hands article, and while the final commercial posted a very slight improvement, it’s still massively underwhelming compared to what we expected.

The video test is both in part a test of CPU responsiveness as well as SoC decoder/encoder as well as OS API tests – with a mix of I/O performance thrown in. Flagship devices in this test have been relatively grouped tightly together for some time now but the Snapdragon 845 still manages a slight lead over the Exynos 9810 Galaxy S9.

The Photo Editing test uses OS APIs to apply common effects on pictures, which in turn make use of RenderScript to enable GPU acceleration. The Qualcomm Galaxy S9 again posts excellent performance here, while not quite matching the promised performance on the QRD845, it still leads all other commercial devices.

Huawei’s Kirin SoCs use the same Mali GPU in an even weaker configuration, yet they showcase over double the performance. The issue here on Samsung SoCs again seems to be software related as the Renderscript are relatively short which makes the GPU never goes past the minimum 260MHz. Huawei optimises the DVFS driver to compute workloads and enacts a boost to ensure better performance.

The Writing and Data Manipulation tests are also heavily reliant on OS APIs and make use on Android UI rendering. The Snapdragon 845 here sees a larger discrepancy from the QRD platform, especially in the data manipulation test. The Writing test is especially I feel is a workload that is able to accurately represent the “snappiness” feeling of a device and to date the relative ranking between devices.

The Exynos Galaxy S9 again does not do well in in any of these workloads. I’ll talk about how this translated into real world performance in a bit.

The overall performance for PCMark and the Galaxy S9 sees a stark contrast between the Snapdragon and Exynos variants. Both variants don’t show that great of improvements over their predecessors, but the Exynos variant especially has such meagre improvements that it barely manages to distance itself from last year’s Exynos 8895 Galaxy S8 running Android Oreo 8.0 firmware. I also was disappointed in the PCMark performance of the Snapdragon 845 – but at least again this variant of the Galaxy S9 manages to top the rankings of current commercial devices, even if it doesn’t quite match the QRD845 scores.

As an aside, I’m still disappointed that Google restricted Accessibility Events in Android 7 and consequently  broke one of my favourite performance measuring tools; DiscoMark, which we last used in our Galaxy Note7 review. DiscoMark was able to empirically measure application startup times without them having to be compiled with debugging features on – the results were the best we ever had in terms of device responsiveness.

Until I find a replacement to empirically measure responsiveness, what I can give is my subjective experience with both variants of the Galaxy S9. The Snapdragon variant of the S9 is extremely fast and is up there with Google’s Pixel 2’s and is among the fastest Android devices I’ve used. I’ve got very little to complain about as it performed superbly. The Exynos 9810 was equally an extremely fast device, and make no mistake it as was among one of the fastest devices out there, however I found that it didn’t quite match the UI responsiveness of the Snapdragon variant in some scenarios. I also recently upgraded my Exynos 8895 Galaxy S8 to Android 8.0 Oreo and that seemed to have improved responsiveness when switching between apps, which further closed the gap between it and the Exynos 9810 variant of the S9.

Continuing onto our latest set of browser benchmarks we start with our new adoption of Speedometer 2.0 which is meant to replace past JavaScript micro-benchmarks with more representative JS framework tests showcasing web UI responsiveness. The benchmark is backed by the WebKit team at Apple and fully endorsed by Google’s Chrome team.

The Snapdragon Galaxy S9 sees the excellent boost that we saw on Qualcomm’s reference platform and even actually manages to beat that score, leading all Android devices. Apple’s iPhone 7 and iPhone 8 generations are still ahead – this is due to Apple’s much faster Nitro JS engine which keeps improving, but also thanks to the A- chipset’s excellent CPUs which do great work in terms of raw performance.

The Exynos 9810 Galaxy S9 posts a massively disappointing score and just barely manages to outperform last year’s Galaxy S8’s by a hair’s margin. Certainly when I first heard of Samsung’s new big CPU I expected finally to have a SoC manage to compete in Apple in performance, however what we’re seeing here is just bad.

I hinted out that we would be switching to WebXPRT 3 for our 2018 suite and PrincipledTechnolgies released the new benchmark in late February. This will be the last article including WebXPRT 2015 as we compare the relative ranking of devices in both benchmarks.

The Galaxy S9 in its Snapdragon variant sees excellent gains in both benchmarks and puts a clear generational gap between it and past Android devices, and actually manages to outperform the iPhone 7 in this test.

On the Exynos side of things we see again the trend of disappointing scores as this variant of the Galaxy S9 cannot distinguish itself against an Android 8.0 running variant of the E8895 Galaxy S8, going as far to scoring less than both S8’s in WebXPRT 2015.

Are the benchmarks correct, and why are they like that?

AnandTech is usually data-driven when making claims about performance so the stark contrast between the Exynos’ synthetic performance and the system benchmarks more than ever question the validity of both. There are two questions to answer here: are the benchmarks still working as intended and representative, and if they are, what happened to the Exynos 9810’s raw performance?

For the first question, I haven’t seen any evidence to contradict the results of our system benchmarks. The Exynos 9810 variant of the Galaxy S9 simply isn’t any faster in most workloads and in one-on-one comparisons against the Snapdragon 845 variant it was indeed the less consistent one in performance and losing out in terms of responsiveness, even if that difference in absolute terms is very minor.

As to why this is happening on the Exynos is something that I attribute to scheduler and DVFS. Samsung’s new scheduler and DVFS just seems absolutely atrociously tuned for performance. I tested an interactive workload on both Snapdragon and Exynos devices and the contrast couldn’t be any greater. On the Snapdragon 845 Galaxy S9 a steady state workload thread will seemingly migrate from a full idle state of the little CPUs onto the big CPUs after 65ms. At the migration moment the big CPUs kick into full gear at 2803MHz and will maintain that frequency for as long as the workload demands it.

On the Exynos 9810 Galaxy S9 the same workload will also migrate around at the 60ms time from the little cores up to the big cores, however once on the big cores the thread starts at the lowest frequencies of the big cluster – 650-741MHz. It takes the workload a whole 370ms until it reaches the 2314MHz state of the M3 cores – which according to the SPEC benchmarks is around the maximum single-threaded performance of the Snapdragon 845’s performance cores. To reach the full 2703MHz of the M3 cores the workload needs to have been active for a staggering 410ms before the DVFS mechanism starts switching to that frequency state.

UI workloads are highly transactional and very rarely is there something which takes longer than a few frames. The fact that the Exynos 9810 takes over 5x longer to reach the maximum performance state of the Snapdragon 845 basically invalidates absolutely everything about the performance of its cores. For workloads which are shorter than 400ms (which is a *lot* of time in computing terms) the Snapdragon will have already finished the race before the Exynos warms up. Only at higher workload durations would the Exynos then finally catch up. Acceleration vs maximum speed being the key aspects here. This is Samsung’s first EAS based scheduler for Exynos devices, and the way the schedutil governor is tuned here is a great disappointment for performance.

Beyond the Exynos’ overzealous “slow-and-steady” DVFS approach I’m also not happy how the core count/maximum frequency mechanism is implemented. This is a simple HR timer task that checks the CPU runqueues and based on a threshold of heavy threads it simply offlines or onlines the CPUs. The fixed interval here is 15ms when in a quad-core state and 30ms in dual- and single-core states. Beyond the fact that the whole offlineing/onlineing of the cores is extremely inefficient as a scheduler mechanism, it’s worrisome that when the SoC is in dual or single-mode and there’s suddenly a burst of threads, the CPUs will be highly underprovisioned in terms of capacity up to 30ms until the mechanism turns back on the other cores.