Performance Consistency

We've been looking at performance consistency since the Intel SSD DC S3700 review in late 2012 and it has become one of the cornerstones of our SSD reviews. Back in the days many SSD vendors were only focusing on high peak performance, which unfortunately came at the cost of sustained performance. In other words, the drives would push high IOPS in certain synthetic scenarios to provide nice marketing numbers, but as soon as you pushed the drive for more than a few minutes you could easily run into hiccups caused by poor performance consistency. 

Once we started exploring IO consistency, nearly all SSD manufacturers made a move to improve consistency and for the 2015 suite, I haven't made any significant changes to the methodology we use to test IO consistency. The biggest change is the move from VDBench to Iometer 1.1.0 as the benchmarking software and I've also extended the test from 2000 seconds to a full hour to ensure that all drives hit steady-state during the test.

For better readability, I now provide bar graphs with the first one being an average IOPS of the last 400 seconds and the second graph displaying the IOPS divided by standard deviation during the same period. Average IOPS provides a quick look into overall performance, but it can easily hide bad consistency, so looking at standard deviation is necessary for a complete look into consistency.

I'm still providing the same scatter graphs too, of course. However, I decided to dump the logarithmic graphs and go linear-only since logarithmic graphs aren't as accurate and can be hard to interpret for those who aren't familiar with them. I provide two graphs: one that includes the whole duration of the test and another that focuses on the last 400 seconds of the test to get a better scope into steady-state performance. These results are for all drives 240GB and up.

Steady-State 4KB Random Write Performance

Steady-state performance is fairly good, although given that the SX930 employes 12% over-provisioning it's expected to perform better than drives with only 7%. 

Steady-State 4KB Random Write Consistency

Consistency, on the other hand, isn't that good. The good news is that the JMF670H is more consistent than the SM2246EN in the BX100, but there's a long way to achieve Samsung-level consistency. 

25% Over-Provisioning

As the SX930 isn't very consistent, we saw performance dropping below 1,000 IOPS. It seems that consistency is one of the areas that truly separate Samsung and Marvell drives from the rest because JMicron, Silicon Motion and Phison based drives all have trouble sustaining steady performance. Basically, the baseline performance is about 1,000 IOPS for all controllers with frequent peaks of +5,000 IOPS, whereas Samsung and Marvell drives have very little variation in performance. I suspect the controller design itself has something to do with this because Samsung and Marvell controllers are all multi-core, but JMicron and Silicon Motion rely on single-core designs for higher cost efficiency. 

25% Over-Provisioning
Introduction, The Drives & The Test AnandTech Storage Bench - The Destroyer
Comments Locked


View All Comments

  • dada121 - Thursday, July 16, 2015 - link

    First to take the throne,
  • Refuge - Thursday, July 16, 2015 - link

    can't wait for NVMe to give us some space to stretch our legs again! :)
  • Stochastic - Thursday, July 16, 2015 - link

    Just how much of an impact are NVMe drives expected to have on light tasks, e.g. boot times, application and game load times, etc. Could the average consumer benefit from a move to ultra high performance NVMe drives, or are the benefits limited to power users?
  • Refuge - Thursday, July 16, 2015 - link

    It is definitely more limited to power users. Especially until NVMe support and drives go down in costs. Right now it is enterprise or enthusiast only.

    But that is just par for the course. Software will be developed to use the extra horsepower one day, but not until after that level of performance is much more common place.

    SATA6 SSD's will be perfect for regular consumers for years to come. But I see in a couple years especially with how graphics resolutions are going through the roof that Gamers will start finding a use for the extra bandwidth once 4k gaming is mainstream and texture packs start to explode like they did back in the 90's.

    Remember when a 16g HDD running at 5400 RPM was the SHIT!?
  • TelstarTOS - Tuesday, July 21, 2015 - link

    Yeah, in those times i had SCSI :)
  • Adding-Color - Thursday, July 16, 2015 - link

    They could be useful for video editing and other stuff that profits from fast sequential reads/writes.
    For games not so much. Most games appear to be CPU limited not SSD disk limited (for load times, when you are using a SSD) and a recent review (forgot the link) showed almost no load times improvement using a 2GB/s NVME Pcie SSD compared to a 500MB/s SATA SSD.

    From a power and and efficiency standpoint NVME should have less latency and less lower consumption.
  • Impulses - Thursday, July 16, 2015 - link

    Content creation tasks, even with photos instead of video, can already benefit greatly from NVMe or M2/PCI-E in general... I'm getting a smaller M2 Samsung drive as soon as I have a mobo that supports it (Skylake?), not sure I see myself going for a large one until prices drop well under $1/GB tho, and that'll take a while.

    For a power user I think an M2/PCI-E SSD for OS/apps/scratch space + large SATA SSD drive/array for data will soon supplant the SATA SSD + HDD setups... But M2/PCI-E will be like the Raptors of the SSD world.
  • AnnonymousCoward - Thursday, July 16, 2015 - link

    Stochastic, you won't find the answer here, since AT doesn't do real world SSD testing.
  • benzosaurus - Sunday, July 19, 2015 - link

    Half of their benchmarks are literally recordings of the writes generated by doing real world things.
  • AnnonymousCoward - Sunday, July 19, 2015 - link

    Irrelevant. Stochastic asked how boot times and app load times compare.

Log in

Don't have an account? Sign up now