CPU Tests: Office and Science

Our previous set of ‘office’ benchmarks have often been a mix of science and synthetics, so this time we wanted to keep our office section purely on real world performance.

Agisoft Photoscan 1.3.3: link

The concept of Photoscan is about translating many 2D images into a 3D model - so the more detailed the images, and the more you have, the better the final 3D model in both spatial accuracy and texturing accuracy. The algorithm has four stages, with some parts of the stages being single-threaded and others multi-threaded, along with some cache/memory dependency in there as well. For some of the more variable threaded workload, features such as Speed Shift and XFR will be able to take advantage of CPU stalls or downtime, giving sizeable speedups on newer microarchitectures.

For the update to version 1.3.3, the Agisoft software now supports command line operation. Agisoft provided us with a set of new images for this version of the test, and a python script to run it. We’ve modified the script slightly by changing some quality settings for the sake of the benchmark suite length, as well as adjusting how the final timing data is recorded. The python script dumps the results file in the format of our choosing. For our test we obtain the time for each stage of the benchmark, as well as the overall time.

(1-1) Agisoft Photoscan 1.3, Complex Test

Application Opening: GIMP 2.10.18

First up is a test using a monstrous multi-layered xcf file to load GIMP. While the file is only a single ‘image’, it has so many high-quality layers embedded it was taking north of 15 seconds to open and to gain control on the mid-range notebook I was using at the time.

What we test here is the first run - normally on the first time a user loads the GIMP package from a fresh install, the system has to configure a few dozen files that remain optimized on subsequent opening. For our test we delete those configured optimized files in order to force a ‘fresh load’ each time the software in run. As it turns out, GIMP does optimizations for every CPU thread in the system, which requires that higher thread-count processors take a lot longer to run.

We measure the time taken from calling the software to be opened, and until the software hands itself back over to the OS for user control. The test is repeated for a minimum of ten minutes or at least 15 loops, whichever comes first, with the first three results discarded.

(1-2) AppTimer: GIMP 2.10.18

GIMP as a test can be confusing, but it's best to look at it as a single threaded test but the number of loops required is a function of the thread count. Fast small core count chips do best here.


In this version of our test suite, all the science focused tests that aren’t ‘simulation’ work are now in our science section. This includes Brownian Motion, calculating digits of Pi, molecular dynamics, and for the first time, we’re trialing an artificial intelligence benchmark, both inference and training, that works under Windows using python and TensorFlow.  Where possible these benchmarks have been optimized with the latest in vector instructions, except for the AI test – we were told that while it uses Intel’s Math Kernel Libraries, they’re optimized more for Linux than for Windows, and so it gives an interesting result when unoptimized software is used.

3D Particle Movement v2.1: Non-AVX and AVX2/AVX512

This is the latest version of this benchmark designed to simulate semi-optimized scientific algorithms taken directly from my doctorate thesis. This involves randomly moving particles in a 3D space using a set of algorithms that define random movement. Version 2.1 improves over 2.0 by passing the main particle structs by reference rather than by value, and decreasing the amount of double->float->double recasts the compiler was adding in.

The initial version of v2.1 is a custom C++ binary of my own code, and flags are in place to allow for multiple loops of the code with a custom benchmark length. By default this version runs six times and outputs the average score to the console, which we capture with a redirection operator that writes to file.

For v2.1, we also have a fully optimized AVX2/AVX512 version, which uses intrinsics to get the best performance out of the software. This was done by a former Intel AVX-512 engineer who now works elsewhere. According to Jim Keller, there are only a couple dozen or so people who understand how to extract the best performance out of a CPU, and this guy is one of them. To keep things honest, AMD also has a copy of the code, but has not proposed any changes.

The 3DPM test is set to output millions of movements per second, rather than time to complete a fixed number of movements.

(2-1) 3D Particle Movement v2.1 (non-AVX)(2-2) 3D Particle Movement v2.1 (Peak AVX)

y-Cruncher 0.78.9506: www.numberworld.org/y-cruncher

If you ask anyone what sort of computer holds the world record for calculating the most digits of pi, I can guarantee that a good portion of those answers might point to some colossus super computer built into a mountain by a super-villain. Fortunately nothing could be further from the truth – the computer with the record is a quad socket Ivy Bridge server with 300 TB of storage. The software that was run to get that was y-cruncher.

Built by Alex Yee over the last part of a decade and some more, y-Cruncher is the software of choice for calculating billions and trillions of digits of the most popular mathematical constants. The software has held the world record for Pi since August 2010, and has broken the record a total of 7 times since. It also holds records for e, the Golden Ratio, and others. According to Alex, the program runs around 500,000 lines of code, and he has multiple binaries each optimized for different families of processors, such as Zen, Ice Lake, Sky Lake, all the way back to Nehalem, using the latest SSE/AVX2/AVX512 instructions where they fit in, and then further optimized for how each core is built.

For our purposes, we’re calculating Pi, as it is more compute bound than memory bound. In single thread mode we calculate 250 million digits, while in multithreaded mode we go for 2.5 billion digits. That 2.5 billion digit value requires ~12 GB of DRAM, and so is limited to systems with at least 16 GB.

(2-3) yCruncher 0.78.9506 ST (250m Pi)(2-4) yCruncher 0.78.9506 MT (2.5b Pi)

NAMD 2.13 (ApoA1): Molecular Dynamics

One of the popular science fields is modeling the dynamics of proteins. By looking at how the energy of active sites within a large protein structure over time, scientists behind the research can calculate required activation energies for potential interactions. This becomes very important in drug discovery. Molecular dynamics also plays a large role in protein folding, and in understanding what happens when proteins misfold, and what can be done to prevent it. Two of the most popular molecular dynamics packages in use today are NAMD and GROMACS.

NAMD, or Nanoscale Molecular Dynamics, has already been used in extensive Coronavirus research on the Frontier supercomputer. Typical simulations using the package are measured in how many nanoseconds per day can be calculated with the given hardware, and the ApoA1 protein (92,224 atoms) has been the standard model for molecular dynamics simulation.

Luckily the compute can home in on a typical ‘nanoseconds-per-day’ rate after only 60 seconds of simulation, however we stretch that out to 10 minutes to take a more sustained value, as by that time most turbo limits should be surpassed. The simulation itself works with 2 femtosecond timesteps. We use version 2.13 as this was the recommended version at the time of integrating this benchmark into our suite. The latest nightly builds we’re aware have started to enable support for AVX-512, however due to consistency in our benchmark suite, we are retaining with 2.13. Other software that we test with has AVX-512 acceleration.

(2-5) NAMD ApoA1 Simulation

AI Benchmark 0.1.2 using TensorFlow: Link

Finding an appropriate artificial intelligence benchmark for Windows has been a holy grail of mine for quite a while. The problem is that AI is such a fast moving, fast paced word that whatever I compute this quarter will no longer be relevant in the next, and one of the key metrics in this benchmarking suite is being able to keep data over a long period of time. We’ve had AI benchmarks on smartphones for a while, given that smartphones are a better target for AI workloads, but it also makes some sense that everything on PC is geared towards Linux as well.

Thankfully however, the good folks over at ETH Zurich in Switzerland have converted their smartphone AI benchmark into something that’s useable in Windows. It uses TensorFlow, and for our benchmark purposes we’ve locked our testing down to TensorFlow 2.10, AI Benchmark 0.1.2, while using Python 3.7.6.

The benchmark runs through 19 different networks including MobileNet-V2, ResNet-V2, VGG-19 Super-Res, NVIDIA-SPADE, PSPNet, DeepLab, Pixel-RNN, and GNMT-Translation. All the tests probe both the inference and the training at various input sizes and batch sizes, except the translation that only does inference. It measures the time taken to do a given amount of work, and spits out a value at the end.

There is one big caveat for all of this, however. Speaking with the folks over at ETH, they use Intel’s Math Kernel Libraries (MKL) for Windows, and they’re seeing some incredible drawbacks. I was told that MKL for Windows doesn’t play well with multiple threads, and as a result any Windows results are going to perform a lot worse than Linux results. On top of that, after a given number of threads (~16), MKL kind of gives up and performance drops of quite substantially.

So why test it at all? Firstly, because we need an AI benchmark, and a bad one is still better than not having one at all. Secondly, if MKL on Windows is the problem, then by publicizing the test, it might just put a boot somewhere for MKL to get fixed. To that end, we’ll stay with the benchmark as long as it remains feasible.

(2-6) AI Benchmark 0.1.2 Total

As mentioned, AI-Benchmark is not really optimized at all for Windows, with some issues in the threading due to MKL. AMD seems to perform better here, but it's an interesting mix between chiplets and cores vs frequency.

CPU Tests: Microbenchmarks CPU Tests: Simulation


View All Comments

  • edzieba - Monday, January 4, 2021 - link

    I dunno, sounds like an opportunity for ambient-pressure water phase-change cooling to me! Who needs evacuated heat-pipes or vapour-chambers when you can just spray the top of the IHS directly! Reply
  • shabby - Monday, January 4, 2021 - link

    Hey Ian can you put the real cpu wattage in the charts that the cpu used in that test rather than the fake one? We all know this cpu never uses 125 watts. Reply
  • Drkrieger01 - Monday, January 4, 2021 - link

    You either skipped the 'Power Consumption' page, or don't understand CPU TDP ratings. The '125W' rating is the 'non-turbo' rating, meaning power consumed at max non-turbo clock rate. AMD does the same thing, and also has a higher power consumption during turbo (although not anywhere near as much as Intel does). Reply
  • shabby - Monday, January 4, 2021 - link

    Since each benchmark varies it would be nice seeing how much wattage each cpu used during that benchmark.
    Yes i know amd uses more power during turbo, the 5950x uses 30 watts more than advertised... compared to ~140 watts more that intel advertises their 10850k to use. That quite the difference don't you think?
  • Drkrieger01 - Monday, January 4, 2021 - link

    Unless you're working on a power budget, I honestly wouldn't worry about it. Most review websites don't have the time/man-power to trace the power usage on each benchmark for each CPU. You will also have a variance between processors of the exact same model due to binning/silicon lottery. You're better off planning to use/dissipate the full turbo power of the CPU than hope for lower power. Or just buy an AMD (if you can find one!) Reply
  • eek2121 - Monday, January 4, 2021 - link

    Actually AMD chips use the TDP value as the maximum power value minus the IO power, so all AMD chips use a total of 143 watts at maximim. Reply
  • npz - Tuesday, January 5, 2021 - link

    AMD does NOT do the "same thing", not even close. Even the article states the two measure it differently. AMD's TDP has always stuck very close to actual consumption even with turbo modes, at most +10% for in-core temps/power. Full package temps for the IF and I/O chiplets consume a bit more. Reply
  • Flunk - Monday, January 4, 2021 - link

    Intel seems to have six similar i9 SKUs with prices ranging from $453 to $488. Seems rather pointless. Maybe Intel marketing should spend some time thinking about whether or not their insanely complex model scheme is contributing to their lack of sales. AMD has ONE SKU that competes with all of those Intel SKUs. Clock down for lower TDP doesn't need to be an entire SKU. Reply
  • Duwelon - Monday, January 4, 2021 - link

    Whoever comes up with Intel's SKUs must be the same person/people responsible for interfacing with USB Implementers Forum on Intel's behalf. The industry is replete with remarkably confusing naming schemes, seemingly on purpose. Reply
  • DanNeely - Monday, January 4, 2021 - link

    Making the low power versions use the same model number would be a very anti-consumer move because you'd have no easy way to know if you were getting the 3.7Ghz or 1.9Ghz model. We already have that problem on mobile where two laptops with identical specs perform wildly different because one is running the CPU at 2x the power/performance of the other. Using separate model numbers also lets you bin chips that perform best at low and high power levels separately.

    The production limit bins (10850K and both IGPless KF models) muddle things up a bit; but Intel's desktop lines are very cleanly broken out vs what they did a decade+ ago with a mess of different similar chips with varying cache sizes and clock speeds but the same core counts; or the ongoing mess of their mobile line (good luck figuring anything out about one of those chips from its model number without looking it up).

Log in

Don't have an account? Sign up now