Intel's new 3D XPoint non-volatile memory technology, which has been on the cards publically for the last couple of years, is finally hitting the market as the storage medium for Intel's new flagship enterprise storage platform. The Intel Optane SSD DC P4800X is a PCIe SSD using the standard NVMe protocol, but the use of 3D XPoint memory instead of NAND flash memory allows it to deliver great throughput and much lower access latency than any other NVMe SSD.

3D XPoint

The potential significance of 3D XPoint memory is immense. When it was first publicly announced by Intel and Micron in 2015, 3D XPoint memory was a fundamentally different storage technology from the flash memory that dominates the market. It is the first new truly mass market, high-density solid state storage medium to hit the market since NAND flash itself. It comes at a time where the NAND market is booming like never before, but also at a time when we know that there is a definite end of the line for NAND. The ongoing transition to 3D NAND flash is just a temporary postponement of the fundamental limitations of flash memory. Once NAND can no longer scale in density and cost-per-bit, it will fall to paradigm changes and next-generation memory technologies (one of which will be 3D XPoint) to continue to carry the industry forward. There are many other new memory technologies that may compete alongside flash memory and 3D XPoint in the coming years, but 3D XPoint is the one that's ready to go mainstream now.

In the near term, 3D XPoint is important because it offers a new set of performance tradeoffs entirely unlike NAND; tradeoffs that, for the right applications, can deliver performance far in excess of today's NAND products. By being able to read and write at the bit or word level - and not the 4K+ page level of NAND - 3D XPoint has the potential to deliver excellent performance across a wide range of workloads, but especially in minimally parallel workloads, which are common in the consumer and enterprise spaces.

The drawback here is that, due to various factors regarding time, production, and scope, 3D XPoint is more expensive than NAND. It also comes in as less dense, to aid in ease of production in this first stage, but this also adds to the cost. For now, due to scale and other factors, it won't be able to replicate the sheer capacity and cost effectiveness that has made NAND storage so popular in all market segments. Due to the scale, especially as a first-generation version of the technology, the first 3D XPoint products are being aimed at speciality and high-margin markets: enterprise performance, consumer caching, etc. Future products promised from Intel should add non-volatile DIMMs to the mix, and then later on, if everything goes to plan, a potential wholesale replacement of NAND flash (or at least a strong competitor).

The Intel Optane SSD DC P4800X

The new storage drive, and the focus of today's review, is the Intel Optane SSD DC P4800X. It uses a new NVMe controller Intel developed specifically for use with 3D XPoint memory. Where Intel's enterprise NVMe SSDs like the P3700 use a controller with 18 channels for interfacing to their flash memory, the Optane SSD's controller has only 7 channels. In order to achieve at least parity on peak performance, each of those channels has to provide much higher throughput than on a flash SSD, and it shows that each 3D XPoint memory die is delivering much higher performance than a die of flash memory.

The first capacity of the Optane SSD DC P4800X to ship and the model we've tested here offers a usable capacity of 375GB from a total of 28 3D XPoint memory dies (four per channel) for a raw capacity of 448GB. 3D XPoint memory has better endurance than NAND flash, but not enough to get away without wear levelling. The fine-grained accessibility of 3D XPoint memory gets rid of a lot of the wear leveling and write amplification headaches caused by flash pages and erase blocks being larger than the sector sizes exposed by the drives, but the drive still needs some spare area plus storage for error correction overhead, metadata for tracking the mapping between logical blocks and physical addresses, and potential replacement of bad sectors, similar to normal SSDs.

As with most NVMe SSDs, the Optane SSD DC P4800X supports a configurable sector size. Out of the box it emulates 512B sectors for the sake of compatibility, but using the NVMe FORMAT command it can be switched to emulate 4kB sectors. The larger sector size reduces the amount of metadata the SSD controller has to juggle, so it usually allows for slightly higher performance. The NVMe FORMAT command is also the mechanism for triggering a secure erase of the entire drive, and for flash SSDs the format usually consists of little more than issuing block erase commands to the whole drive. 3D XPoint memory does not have large multi-megabyte erase blocks, so a low-level format of the Optane SSD needs to directly write to the entire drive, which takes about as long as filling it sequentially. Thus, while a 2.4TB flash SSD can perform a low-level format in just over 13 seconds, the 375GB Optane SSD DC P4800X takes six minutes and 47 seconds. This is long enough that unsuspecting software tools or SSD reviewers will give up and assume that the drive has locked up.

Intel Optane SSD DC P4800X Specifications
Capacity 375 GB 750 GB 1.5 TB
Form Factor PCIe HHHL or 2.5" 15mm U.2
Interface PCIe 3.0 x4 NVMe
Controller Intel unnamed
Memory 128Gb 20nm Intel 3D XPoint
Typical Latency (R/W) <10µs
Random Read (4 kB) IOPS (QD16) 550,000 TBA TBA
Random Read 99.999% Latency (QD1) 60µs TBA TBA
Random Read 99.999% Latency (QD16) 150µs TBA TBA
Random Write (4 kB) IOPS (QD16) 500,000 TBA TBA
Random Write 99.999% Latency (QD1) 100µs TBA TBA
Random Write 99.999% Latency (QD16) 200µs TBA TBA
Mixed 70/30 (4kB) Random IOPS (QD16) 500,000 TBA TBA
Endurance 30 DWPD
Warranty 5 years (3 years during early limited release)
MSRP $1520 TBA TBA
Release Date HHHL March 19  Q2 2017 2H 2017
U.2 Q2 2017 2H 2017

So far, Intel has only started shipping the 375GB Optane SSD DC P4800X to select customers, and they have not released detailed specifications for the larger capacities that will ship later this year.

It is worth noting that the performance specifications for the P4800X, as provided in the product specification sheets, cover a different set of metrics than Intel usually reports for their enterprise SSDs. Sequential performance is not mentioned at all, but the product brief has quite a bit to say about latency: average latency for QD1 reads and writes, and 99.999th percentile latency for both reads and writes at QD1 and QD16. The fact that Intel is publishing a five-nines QoS metric at all suggests that they plan to set a new standard for performance consistency.

The throughput claims are also remarkable: half a million IOPS or more for reads, writes and a 70/30 read/write mix. There are already drives on the market that can deliver more than 550k random read IOPS, but those SSDs are far larger than 375GB and they require very high queue depths to hit 550k IOPS. There are even a few multi-TB drives that can beat 500k random write IOPS, but they can't sustain that performance indefinitely. The Optane SSD DC P4800X is promising an unprecedented level of storage performance both in absolute terms and relative to its capacity, so it is interesting to see where Intel is going to lay down its line in the sand.

The P4800X will not really occupy the same niche as the multi-TB monsters that offer comparable throughput. With limited capacity but the highest level of performance, this Optane SSD most closely fits the role of SLC NAND based SSDs. SLC has disappeared from the SSD market as virtually all customers preferred to sacrifice a little bit of performance to double their capacity by using MLC NAND. One of the last high-performance SLC SSDs was the Micron P320h, a PCIe SSD from 2012 that slightly pre-dated NVMe and used 34nm SLC NAND flash. Anyone still using a P320h for its consistent low latency performance will be very interested in the P4800X. Outside of that niche, the Optane SSD will obviously be desirable for its raw throughput, but the low capacity may be problematic for some use cases.

One of the unique and most notable performance advantages of the Optane SSD DC P4800X is that it does not require extremely high queue depths to reach full throughput. Enterprise customers have long had to design their systems around the fact that getting full performance out of the fastest PCIe SSDs requires loading them down with queue depths of 128 or higher, sometimes requiring applications to use dozens of threads for I/O. In the client space achieving such queue depths is outright impossible, and in the enterprise space it doesn't happen for free. The P4800X's high performance at low queue depths makes it a much easier drive to get great real-world performance out of.

Intel originally introduced 3D XPoint memory as having far higher write endurance than NAND flash—on the order of 1000x higher. The Optane SSD DC P4800X is rated for 30 drive writes per day (DWPD) for five years, and the current models shipping during this early limited availability period are only rated for three years, rather than the five years it expects the support for the full retail models. Intel says they're being extremely conservative with a new and unproven technology, and doing the math means that 30 DWPD doesn't provide any endurance advantage over the most highly over-provisioned flash-based enterprise SSDs. In terms of total petabytes written, the P4800X only has four-fifths the endurance of the SLC-based Micron P320h. Even allowing for Intel's original comparisons possibly having been relative to lower-endurance contemporary MLC or TLC flash, it seems like this first generation of 3D XPoint memory is not as durable as originally planned - the headline number of 30 DWPD is aimed at alleviating that issue, however for Intel to match its original intentions then the second and third generation parts will have to be a step up, and we look forward to testing them.

Pricing

The MSRP for the 375GB P4800X is $1520, though it will be quite some time before it can readily be ordered from major online retailers. At slightly more than $4/GB, the P4800X will be almost twice as expensive per GB as Intel's next most pricey SSD, the P3608 (which is really two drives in one plus a PCIe switch). Compared to Intel's fastest single SSD (the P3700), the P4800X will be more than three times as expensive per GB. In the broader SSD market, $4/GB is not completely unprecedented, but most companies selling drives in this price range don't even pretend to have a retail price.

This Review

For this review of the Intel Optane SSD DC P4800X, first, we are going to take a deeper dive into what 3D XPoint actually is. Then we go through our testing suite for enterprise drives, testing Intel's claims on performance.

It is worth noting that there is no such thing as a general-purpose enterprise SSD. Enterprise storage workloads are far more varied than client workloads and it is impossible to make general statements about whether random or sequential performance is more important, what kind of mix of reads and writes to expect, or what queue depth is apporpriate to test with. Real-world application benchmarks are difficult to construct and typically end up being far more narrowly applicable than we would hope. Our strategy for this review is to provide a very broad range of synthetic tests with the knowledge that not all results will be relevant to all use cases. Enterprise customers must know and understand their own workload. Since this is our first time testing anything with 3D XPoint memory, this review includes some new benchmarks that would probably not be applicable to a flash SSDs review, making for some interesting numbers.

3D XPoint Refresher
Comments Locked

117 Comments

View All Comments

  • ddriver - Friday, April 21, 2017 - link

    *450 ns, by which I mean lower by 450 ns. And the current xpoint controller is nowhere near hitting the bottleneck of PCIE. It would take a controller that is at least 20 times faster than the current one to even get to the point where PCIE is a bottleneck. And even faster to see any tangible benefit from connecting xpoint directly to the memory controller.

    I'd rather have some nice 3D SLC (better than xpoint in literally every aspect) on PCIE for persistent storage RAM in the dimm slots. Hyped as superior, xpoint is actually nothing but a big compromise. Peak bandwidth is too low even compared to NVME NAND, latency is way too high and endurance is way too low for working memory. Low queue depths performance is good, but credit there goes to the controller, such a controller will hit even better performance with SLC nand. Smarter block management could also double the endurance advantage SLC already has over xpoint.
  • mdriftmeyer - Saturday, April 22, 2017 - link

    ddriver is spot on. just to clarify an early comment. He's correct and the IntelUser2000 is out of his league.
  • mdriftmeyer - Saturday, April 22, 2017 - link

    Spot on.
  • tuxRoller - Friday, April 21, 2017 - link

    We don't know how much slower the media is than dram right now.
    We know than using dram over nvme has similar (though much better worst case) perf to this.
    See my other post regarding polling and latency.
  • bcronce - Saturday, April 22, 2017 - link

    Re-reading, I see it says "typical" latency is under 10us, placing it in spitting distance of DDR3/4. It's the 99.9999th percentile that is 60us for Q1. At Q16, 99.999th percentile is 140us. That means it takes only 140us to service 16 requests. That's pretty much the same as 10us.

    Read Q1 4KiB bandwidth is only about 500MiB/s, but at Q8, it's about 2GiB which puts it on par with DDR4-2400.
  • ddriver - Saturday, April 22, 2017 - link

    "placing it in spitting distance of DDR3/4"

    I hope you do realize that dram latency is like 50 NANOseconds, and 1 MICROsecond is 1000 NANOseconds.

    So 10 us is actually 200 times as much as 50 ns. Thus making hypetane about 200 times slower in access latency. Not 200%, 200X.
  • tuxRoller - Saturday, April 22, 2017 - link

    Yes, the dram media is that fast but when it's exposed through nvme it has the latency characteristics that bcronce described.
  • wumpus - Sunday, April 23, 2017 - link

    That's only on a page hit. For the type of operations that 3dxpoint is looking at (4k or so) you won't find it on an open page and thus take 2-3 times as long till it is ready.

    That still leaves you with ~100x latency. And we are still wondering if losing the PCIe controller will make any significant difference to this number (one problem is that if Intel/Micron magically fixed this, the endurance is only slightly better than SLC and would quickly die if used as main memory).
  • ddriver - Sunday, April 23, 2017 - link

    Endurance for the initial batch postulated from intel's warranty would be around 30k PE cycles, and 50k for the upcoming generation. That's not "only slightly better than SLC" as SCL has 100k PE cycles endurance. But the 100k figure is somewhat old, and endurance goes down with process node. So at a comparable process, SLC might be going down, approaching 50k.

    It remains to be seen, the lousy industry is penny pinching and producing artificial NAND shortages to milk people as much as possible, and pretty much all the wafers are going into TLC, some MLC and why oh why, QLC trash.

    I guess they are saving the best for last. 3D SLC will address the lower density, samsung currently has 2 TB MLC M2, so 1 TB is perfectly doable via 3D SLC. I am guessing samsung's z-nand will be exactly that - SLC making a long overdue comeback.
  • tuxRoller - Sunday, April 23, 2017 - link

    The endurance issue is, imho, the biggest concern right now.

Log in

Don't have an account? Sign up now