Dell R620 and consumer SSDs

[nutcase]

I want to prove a point at work that our database server is being seriously held back running nearline (7.2k) HDDs. I need 1TB of space to do it, and it needs to have a level of redundancy for even a test as it will be live (so all the normal users pushing it to prove it).

So my thoughts are to try and get the OK to buy a couple of 1TB consumer SSDs, put them in caddies and get the people who run the server for us to configure them into a RAID1 array and then move the client facing VM onto them.

But I don't want to spend £600 on SSDs that the server refuses to use. Host system is Hyper-V server 2012 running 2k8 VMs. Two years ago I asked the DB provider if the drives were holding us up - and NOW they've tried to tell me the same thing

The other VMs run background generating tasks so can essentially take as long as they like - it's the client facing server that needs a rocket up its behind.

If I can prove a massive performance increase, getting the budget to put enterprise grade SSDs in would be relatively straightforward.

 

[Uhtred]

Humm I'd build my business case around IOPs, performance logs and best practice documentation. I'm sure you're right, 7.2k HDDs for a busy db is a terrible idea but I wouldn't just throw money at it unless I was sure it was the right thing to do.

Are the logs kept on the same set of spindles? Does it have enough RAM?

 

[DiscoDave]

Quantify and document your findings. If you can prove that the IOPS requirement of your DB server is causing saturation on your 7.2K disks then it's easy to prove.

As a middle ground you could try with 10k/15k SAS drives? I run some pretty hefty DB servers for clients, and we haven't gone all-flash storage yet.

 

[Armageus]

The PERC H710 presumably? RAID card should allow you to use any third party drives - the only issue might be that they are flagged as unsupported in the management software and/or you get warning lights on the drive caddies themselves.

If you can provide some more details, I'm sure some of us can provide more in depth/relevant answers. What database are you running SQL Server/MySQL? How much Ram? What OS? Type of queries e.g. mostly read or mostly write or equally split? Total Database size vs "Hot Data"?

As a middle ground you could try with 10k/15k SAS drives?

With 15k drives tend to be the best compromise, they offer a reasonable step up from 7.2k drives, without the cost and potential "worries" that a full move to SSD bring.

Without knowing more details, it's difficult to say, but if it's a Read focussed app, then maxing the RAM (and if necessary reconfiguring database e.g. buffer pool size on MySQL) would be a good start, to ensure you are hitting the disks as little as possible.

If it's write limited, then throwing hardware at it is probably the easiest option, but things like ensuring transaction logs are on separate drives, and doing batch updates (or grouping via transactions) rather than single updates can reduce your io requirement.

If using SQL Server, then I think there is an option to use an SSD as a cache which might be worth looking into (and could probably be done with "consumer" level SSDs)

 

[nutcase]

The PERC H710 presumably? RAID card should allow you to use any third party drives - the only issue might be that they are flagged as unsupported in the management software and/or you get warning lights on the drive caddies themselves.

Yes it's an H710 - promising that it should at least function. Not too worried about warnings as it would only be temporary.

I have no real ability to build a business case - it's taken 2 years for the people that run it to agree that the drives may be holding it back. We have zero access to the box or the software, and that's not going to change. Changing supplier isn't an option. A foolish position to be in? Certainly

Read/Write is near as dammit 50/50 - managed to get that info. 95th percentile IOPS is a measly 231! Everything on the box runs on the same RAID5 3 disk array

It runs hyper-v 2012 server with Server 2008 R2 for the main database server running MS SQL server. 32GB RAM allocated which is the max for that version of server 2008.

Thanks people

 

[Armageus]

Read/Write is near as dammit 50/50 - managed to get that info. 95th percentile IOPS is a measly 231! Everything on the box runs on the same RAID5 3 disk array

No wonder you are in a bad place, but my worries would be more about reliability than performance!

Rather than go down the SSD route (which potentially has a whole heap of its own problems), I would just move to a more "normal" RAID10 setup of at least 4 Disks (ideally 15k).

A single 15k disk will do 200IOPS, and getting out of RAID5 will remove a lot of overhead (e.g. full stripe + parity calculation for every IO), and put you in a better place reliability wise.


If you still need more performance after this - I would look at Dell/LSI Cachecade - basically putting an SSD in front of your main Disk RAID as a huge cache, and a tiered storage method (like most storage vendors offer), leaving your disk system unchanged.

http://www.dell.com/support/article/uk/en/ukdhs1/SLN156366/EN


I still don't regard SSD has a mature tech at the enterprise level - I know the big players do it, but they have time and money to throw at the problem. Seems like there isn't much information about longevity / performance degradation etc. on SSDs at the SME applications end of the market.

 

[nutcase]

No wonder you are in a bad place, but my worries would be more about reliability than performance!

Rather than go down the SSD route (which potentially has a whole heap of its own problems), I would just move to a more "normal" RAID10 setup of at least 4 Disks (ideally 15k).

A single 15k disk will do 200IOPS, and getting out of RAID5 will remove a lot of overhead (e.g. full stripe + parity calculation for every IO), and put you in a better place reliability wise.


If you still need more performance after this - I would look at Dell/LSI Cachecade - basically putting an SSD in front of your main Disk RAID as a huge cache, and a tiered storage method (like most storage vendors offer), leaving your disk system unchanged.

http://www.dell.com/support/article/uk/en/ukdhs1/SLN156366/EN


I still don't regard SSD has a mature tech at the enterprise level - I know the big players do it, but they have time and money to throw at the problem. Seems like there isn't much information about longevity / performance degradation etc. on SSDs at the SME applications end of the market.

That's interesting as most of what I've read recently is that SSD is the way to go for SQL databases. I have had Dell price up options for 10k, 15k and SSD replacement drives, and whilst the SSDs are eye wateringly expensive, if they make enough difference I would be able to spend the money.

The reason for using a pair of SSDs for a trial is they would be relatively inexpensive to buy, with a couple of caddies off ebay, and would be able to re-task afterwards (probably chuck them in one or other of the directors' laptops to give them a smile!). If I bought a pair of 15k drives and nothing happened - such as the real cause of the system being slow is the config - which wouldn't surprise me - then I've got two 15k drives lying around with nothing I can do with them.

The supplier will only entertain moving an entire VM to a different array, they can't be bothered with putting the log on one and the data file on another.

Helpful I know...

 

[randal]

I'll let you know on Thursday night.

I've got 2x r620s with 2x 1TB Samsung 840 Evos in waiting a reboot and array config.

 

[nutcase]

Oooh

 

[randal]

I've got ESX on those servers, and will be migrating the more I/O intensive VMs over to the SSD.

We're running out of storage space on the mech drives and this was a cost effective solution for increased capacity and performance for the machines that need it. The servers have another 18-24 months warranty on them, and once out of warranty they'll get turned into dev and test scratch boxes so not too worried about the longevity aspect.

Should be able to give you the functionality answers though.

 

[Armageus]

That's interesting as most of what I've read recently is that SSD is the way to go for SQL databases.

I'm not saying they aren't the way to go and things have probably moved on in the 2 Years since I specced our current database servers, but lots of things made me side against them, and unless you you absolutely need the IOPS then I would still advise against them.

More background on how many users, data size or what sort of applications you are using would help build a more compelling argument.

If I bought a pair of 15k drives and nothing happened - such as the real cause of the system being slow is the config - which wouldn't surprise me - then I've got two 15k drives lying around with nothing I can do with them.

Just to clarify, you would need 4 (+ ideally a hotspare as well, so 5), but in return you are going to get at least 4x the performance you get at the minute, if not more due to RAID5 overhead. Reliability will improve measurably (no long rebuild times, no possibility for parity errors, possibility of surviving more than 1 disk failure - as long as in a different mirror set).



I would advise you to look into tuning SQL Server for SSDs, as I suspect you aren't running 2014 which I believe is the first version designed with SSDs in mind, so may be some settings that need changing to stop it assuming it is running on HDDs and tuning appropriately.

If you are dead set on a SSD then there is some background out there worth looking at:
http://blog.xbyte.com/2014/08/14/testing-the-limits-of-the-dell-h710-raid-controller-with-ssd/
http://en.community.dell.com/support-forums/servers/f/906/t/19518324
http://www.modelcar.hk/?p=5933

Cachecade stuff:
http://www.tachytelic.net/2014/05/quick-dell-cachecade-performance-comparison-using-raid-5/
http://www.tachytelic.net/2012/02/a...s-a-cachecade-drive-to-a-dell-poweredge-r510/

 

[Deleted member 138126]

I'm not saying they aren't the way to go and things have probably moved on in the 2 Years since I specced our current database servers, but lots of things made me side against them, and unless you you absolutely need the IOPS then I would still advise against them.

Some of Intel's Enterprise SSDs (e.g. DC S3710) support up to 10 full drive writes per day for the lifetime of the drive. So a 1.2 TB drive will support 12 TB of writes every single day for 5 years (~21 Petabytes over 5 years).

 

[Caged]

I know you've already bought the hardware but have you tried any performance optimisations at all on your database? Throwing your time, scheduled maintenance and new hardware at a problem that could easily be solved by beating your DBA around the head quickly becomes uneconomical.

Some of Intel's Enterprise SSDs (e.g. DC S3710) support up to 10 full drive writes per day for the lifetime of the drive. So a 1.2 TB drive will support 12 TB of writes every single day for 5 years (~21 Petabytes over 5 years).

Which are a different world of performance and pricing from Samsung Evos . Watch these drives like a hawk.

 

[nutcase]

I did find this the other day on the Evos - http://www.vojcik.net/samsung-ssd-840-endurance-destruct-test/ which made me think sod the enterprise grade SSDs, stick with these - but it's the company's data I'd be playing with...

I would *love* to be able to run the proper tests on the server but not going to happen - it's taken 2 years for the suppliers to say their *Might* be a problem We've been saying for over 3 years it's too slow - after about a year they switched the hardware for the R620 that's in there now which helped but wasn't a miracle cure.

I do suspect we'll end up going with 15k drives or possibly 10k as that would still be a roughly 8 or 5 times respectively increase in IOPS. Just want to try SSDs as a test as would only need 2 to prove a performance increase with an IOPS boost.

I am well aware it's utterly the wrong way to be doing things but what choice do we have?

 

[Mark M]

How much input do you have into the decision making? Is it not about time to look to switch providers if that is there attitude?

 

[Armageus]

I did find this the other day on the Evos - http://www.vojcik.net/samsung-ssd-840-endurance-destruct-test/ which made me think sod the enterprise grade SSDs, stick with these - but it's the company's data I'd be playing with...

If it was a test or development server, then I'd say go for it - but given that it's a production server, you don't really want to compromise on anything or leave anything to chance.

I'm not an expert on SQL Server, but I know that up to a certain version on MySQL it was specifically optimised for read/write characteristics of Hard Drives - simply swapping to an SSD could throw up odd conditions e.g. Deadlocks due to some operations being done quicker than it expected.

The reason the Dell SSDs cost more is not even necessarily things like write endurance (as you pointed out the samsungs are pretty good in that respect) but the certification that they won't do anything odd when connected to your raid controller, when run in enclosed/hot spaces like a multi bay server. They are also held in stock and forward/backward compatible, so in 5 years time when you need to replace it, you can buy an exact same model, or one that is functionally identical but with compatible firmware.

I would *love* to be able to run the proper tests on the server but not going to happen - it's taken 2 years for the suppliers to say their *Might* be a problem We've been saying for over 3 years it's too slow - after about a year they switched the hardware for the R620 that's in there now which helped but wasn't a miracle cure.

The fact that they set the drives up in RAID5 is a huge factor in your slow performance - I know your hands are pretty much tied with using them, but this was a terrible choice they made - who knows what other choices they are making if they are e.g. maintaining your SQL Database?

I do suspect we'll end up going with 15k drives or possibly 10k as that would still be a roughly 8 or 5 times respectively increase in IOPS.

getting out of RAID5 with any drives will give you some extra performance:

http://www.dell.com/downloads/global/solutions/tradeoffs_RAID5_RAID10.pdf


Not sure what size your data set is, as to what size drives you need, but £600 would likely buy you 3 out of the 5 drives you would need for RAID10.

I am well aware it's utterly the wrong way to be doing things but what choice do we have?

As long as you know it's not the right way of doing it, then at least you have can have the discussion with management when it all goes wrong.



As a side question, why is RAID1 recommended for SSDs, doesn't seem to make sense, if one drive suffers write endurance failure, or performance degradation, then surely the other one will as well. Other than guarding against a drive completely dying not sure what the benefit is.

 

[nutcase]

How much input do you have into the decision making? Is it not about time to look to switch providers if that is there attitude?

Quite a bit, and we're stuck - there are no alternative suppliers for this type of software

They chose RAID 5 because that's what they did for everyone else. Problem is we're massive compared to their other clients so as soon as we built up to using their system fully (well, we're still not using it fully), it went into meltdown!

 

[Caged]

A 3 disk RAID5 on 7.2k disks for SQL is hilarious.

 

[#Chri5#]

Do you have remote access (RDP etc) to the server?

If so, it should be worth firing up Perfmon to log stats and see if they indicate the storage IO is the bottleneck.

http://blogs.msdn.com/b/sqljourney/...oubleshooting-sql-server-i-o-bottlenecks.aspx
http://www.mssqltips.com/sqlservertip/2329/how-to-identify-io-bottlenecks-in-ms-sql-server/

 

[nutcase]

A 3 disk RAID5 on 7.2k disks for SQL is hilarious.

The more I've read over the last few months makes me agree 100%. I'm not at all familiar with how SQL works.

Unfortunately we have zero access - it's all locked down (their argument is that if we can't access it, we can't break it). It's taken months to get the IOPS info above out of them.