SSD versus Enterprise SAS and SATA disks
by Johan De Gelas on March 20, 2009 2:00 AM EST- Posted in
- IT Computing
Testing in the Real World
As interesting as the SQLIO and IOMeter results are, those benchmarks focus solely on the storage component. In the real world, we care about the performance of our database, mail, or fileserver. The question is: how does this amazing I/O performance translate into performance we really care about like transactions or mails per second? We decided to find out with 64-bit MySQL 5.1.23 and SysBench on SUSE Linux SLES 10 SP2.
We utilize a 23GB database and carefully optimized the my.cnf configuration file. Our goal is to get an idea of performance for a database that cannot fit completely in main memory (i.e. cache); specifically, how will this setup react to the fast SLC SSDs. The Innodb buffer pool that contains data pages, adaptive hash indexes, insert buffers, and locks is set to 1GB. That is indeed rather small, as most servers contain 4GB to 32GB (or even more) and MySQL advises you to use up to 80% of your RAM for this buffer. Our test machine has 8GB of RAM, so we should have used 6GB or more for this buffer. However, we really wanted our database to be about 20 times larger than our buffer pool to simulate a large database that can only partially fit within the memory cache. With our 32GB SLC SSDs, using a 6.5GB buffer pool and a 130GB large database was not an option. Hence, the slightly artificial limitation of our buffer pool size.
We let SysBench perform all kinds of inserts and updates on this 23GB database. As we want to be fully ACID compliant our database is configured with:
innodb_flush_log_at_trx_commit = 1
After each transaction is committed, there is a "pwrite" first followed by an immediate flush to the disk. So the actual transaction time is influenced by the disk write latency even if the disk is nowhere near its limits. That is an extremely interesting case for SSDs to show their worth. We came up with four test configurations:
- "Classical SAS": We use six SAS disks for the data and two for the logs.
- "SAS data with SSD logging": perhaps we can accelerate our database by simply using very fast log disks. If this setup performs much better than "classical SAS", database administrators can boost the performance of their OLTP applications with a small investment in log SSDs. From an "investment" point of view, this would definitely be more interesting than having to replace all your disks with SSDs.
- "SSD 6+2": we replace all of our SAS disks with SSDs. We stay with two SSDs for the logs and six disks for the data.
- "SSD data with SAS logging": maybe we just have to replace our data disks, and we can keep our logging disks on SAS. This makes sense as logging is sequential.
Depending on how many random writes we have, RAID 5 or RAID 10 might be the best choice. We did a test with SysBench on six Intel X25-E SSDs. The logs are on a RAID 0 set of two SSDs to make sure they are not the bottleneck.
As RAID 10 is about 23% faster than RAID 5, we placed the database on a RAID 10 LUN.
Transactional logs are written in a sequential and synchronous manner. Since SAS disks are capable of delivering very respectable sequential data rates, it is not surprising that replacing the SAS "log disks" with SSDs does not boost performance at all. However, placing your database data files on an Intel X25-E is an excellent strategy. One X25-E is 66% faster than eight (!) 15000RPM SAS drives. That means if you don't need capacity, you can replace about 13 SAS disks with one SSD to get the same performance. You can keep the SAS disks as your log drives as they are a relatively cheap way to obtain good logging performance.
Facebook
Twitter
RSS
67 Comments
View All Comments
Rasterman - Monday, March 23, 2009 - link
since the controller is the bottleneck for ssd and you have very fast cpus, did you try testing a full software raid array, just leave the controllers out of it all together?.Snarks - Sunday, March 22, 2009 - link
reading the comments made my brain asplode D:!Damn it, it's way to late for this!
pablo906 - Saturday, March 21, 2009 - link
I've loved the stuff you put out for a long long time. This another piece of quality work. I definitely appreciate the work you put into this stuff. I was thinking about how I was going to build the storage back end for a small/medium virtualization platform and this is definitely swaying some of my previous ideas. It really seems like an EMC enclosure may be in our future instead of a something built by me on a 24 Port Areca Card.I don't know what all the hubub was about at the beginning of the article but I can tell you that I got what I needed. I'd like to see some follow ups in Server Storage and definitely more Raid 6 info. Any chance you can do some serious Raid Card testing, that enclosure you have is perfect for it (I've built some pretty serious storage solutions out of those and 24 port Areca cards) and I'd really like to see different cards and different configurations, numbers of drives, array types, etc. tested.
rbarone69 - Friday, March 20, 2009 - link
Great work on these benchmarks. I have found very few other sources that provided me with the answers to my questions regarding exaclty what you tested here (DETAILED ENOUGH FOR ME). This report will be referenced when we size some of our smaller (~40-50GB but heavily read) central databases we run within our enterprise.It saddens me to see people that simply will NEVER be happy, no matter what you publish to them for no cost to them. Fanatics have their place but generally cost organizations much more than open minded employees willing to work with what they have available.
JohanAnandtech - Saturday, March 21, 2009 - link
Thanks for your post. A "thumbs up" post like yours is the fuel that Tijl and I need to keep going :-). Defintely appreciated!classy - Friday, March 20, 2009 - link
Nice work and no question ssds are truly great performers, but I don't see them being mainstream for several more years in the enterprise world. One is no one knows how relaible they are? They are not tried and tested. Two and three go hand in hand, capapcity and cost. With the need for more and more storage, the cost for ssd makes them somewhat of a one trick pony, a lot of speed, but cost prohibitive. Just at our company we are looking at a seperate data domain just for storage. When you start tallking the need for several terabytes, ssd just isn't going to be considered. Its the future, but until they drastically reduce in cost and increase in capacity, their adoption will be minimal at best. I don't think speed right now trumps capacity in the enterprise world.virtualgeek - Friday, March 27, 2009 - link
They are well past being "untried" in the enterprise - and we are now shipping 400GB SLC drives.gwolfman - Friday, March 20, 2009 - link
[quote]Our Adaptec controller is clearly not taking full advantage of the SLC SSD's bandwidth: we only see a very small improvement going from four to eight disks. We assume that this is a SATA related issue, as eight SAS disks have no trouble reaching almost 1GB/s. This is the first sign of a RAID controller bottleneck.[/quote]I have an Adaptec 3805 (previous generation as to the one you used) that I used to test 4 of OCZ's first SSDs when they came out and I noticed this same issue as well. I went through a lengthy support ticket cycle and got little help and no answer to the explanation. I was left thinking it was the firmware as 2 SAS drives had a higher throughput than the 4 SSDs.
supremelaw - Friday, March 20, 2009 - link
For the sake of scientific inquiry primarily, but not exclusively,another experimental "permutation" I would also like to see is
a comparison of:
(1) 1 x8 hardware RAID controller in a PCI-E 2.0 x16 slot
(2) 1 x8 hardware RAID controller in a PCI-E 1.0 x16 slot
(3) 2 x4 hardware RAID controllers in a PCI-E 2.0 x16 slot
(4) 2 x4 hardware RAID controllers in a PCI-E 1.0 x16 slot
(5) 2 x4 hardware RAID controllers in a PCI-E 2.0 x4 slot
(6) 2 x4 hardware RAID controllers in a PCI-E 1.0 x4 slot
(7) 4 x1 hardware RAID controllers in a PCI-E 2.0 x1 slot
(8) 4 x1 hardware RAID controllers in a PCI-E 1.0 x1 slot
* if x1 hardware RAID controllers are not available,
then substitute x1 software RAID controllers instead,
to complete the experimental matrix.
If the controllers are confirmed to be the bottlenecks
for certain benchmarks, the presence of multiple I/O
processors -- all other things being more or less equal --
should tell us that IOPs generally need more horsepower,
particularly when solid-state storage is being tested.
Another limitation to face is that x1 PCI-E RAID controllers
may not work in multiples installed in the same motherboard
e.g. see Highpoint's product here:
http://www.newegg.com/Product/Product.aspx?Item=N8...">http://www.newegg.com/Product/Product.aspx?Item=N8...
Now, add different motherboards to the experimental matrix
above, because different chipsets are known to allocate
fewer PCI-E lanes even though slots have mechanically more lanes
e.g. only x4 lanes actually assigned to an x16 PCI-E slot.
MRFS
supremelaw - Friday, March 20, 2009 - link
More complete experimental matrix (see shorter matrix above):(1) 1 x8 hardware RAID controller in a PCI-E 2.0 x16 slot
(2) 1 x8 hardware RAID controller in a PCI-E 1.0 x16 slot
(3) 2 x4 hardware RAID controllers in a PCI-E 2.0 x16 slot
(4) 2 x4 hardware RAID controllers in a PCI-E 1.0 x16 slot
(5) 1 x8 hardware RAID controllers in a PCI-E 2.0 x8 slot
(6) 1 x8 hardware RAID controllers in a PCI-E 1.0 x8 slot
(7) 2 x4 hardware RAID controllers in a PCI-E 2.0 x8 slot
(8) 2 x4 hardware RAID controllers in a PCI-E 1.0 x8 slot
(9) 2 x4 hardware RAID controllers in a PCI-E 2.0 x4 slot
(10) 2 x4 hardware RAID controllers in a PCI-E 1.0 x4 slot
(11) 4 x1 hardware RAID controllers in a PCI-E 2.0 x1 slot
(12) 4 x1 hardware RAID controllers in a PCI-E 1.0 x1 slot
MRFS