4Gb fibre channel vs iSCSI over 10GbE

[Shad]

I'm thinking about a new little project to build a SAN involving ZFS, 6x 2TB drives (probably in RAID-Z2) and a nice high speed connection to some other servers. But I'm torn between using some of my existing 10GbE network gear to link it all together and use iSCSI, or get some cheap 4Gb fibre channel HBAs and an FC switch. On the one hand I fancy trying something new, but on the other hand the 10GbE kit has the potential for more bandwidth.

But does it really matter? I don't think I'm likely to be pushing 4Gb FC am I, even with an SSD for L2ARC and ZIL?

Thoughts welcome

 

[RSR]

I run a FC /ZFS SAN at home using Solaris 11 + Comstar which I present to my VMware Node. I can normally completely saturate the link using Crystal Mark when I have bench marked my VMs.

ZFS loves RAM for its ARC, I think the general rule of thumb off the top of my head is 1GB per 1TB.

I am currently running:

ASUS P8B-M
Intel Xeon E3 1230
16GB ECC RAM
LSI 9207-8i HBA
Intel RES2SV240 SAS Exander
Qlogic 4GB 2462 HBA
5 x Seagate 2TB
4 x Intel 335 240GB SSD's
1 x Intel 330 120GB SSD
Plus other bits and bobs

If you have questions please let me know, I did start a project log which I need to update.

 

[Shad]

Thanks Andy. How many drives are you using for parity, just the one? My potential configuration would only allow me to use one SSD, hopefully for the OS install as well as cache but I've read that ZFS needs to use the whole drive rather than a partition? The server has 16GB RAM and 4x 2.33 GHz from dual Xeon 5148, hopefully enough power for ZFS with ~7.5TB usable.

Did you consider things like NexentaStor CE rather than going the DIY route? I'm thinking OpenIndiana and napp-it could be a good way to go for FC, also avoiding the 18TB limit of NexentaStor CE.

Would be good to read your project log if you've got a link

 

[RSR]

Thanks Andy. How many drives are you using for parity, just the one? My potential configuration would only allow me to use one SSD, hopefully for the OS install as well as cache but I've read that ZFS needs to use the whole drive rather than a partition? The server has 16GB RAM and 4x 2.33 GHz from dual Xeon 5148, hopefully enough power for ZFS with ~7.5TB usable.

Did you consider things like NexentaStor CE rather than going the DIY route? I'm thinking OpenIndiana and napp-it could be a good way to go for FC, also avoiding the 18TB limit of NexentaStor CE.

Would be good to read your project log if you've got a link

Here is the link to my Project > Click Here Its changed a bit since then, as I have added more drives & cards.

I have one zpool RAID Z (Raid 5) with a SSD for caching & two RAID 0 pools for my VMware, ZFS likes direct access to the drives. So ideally you would require a HBA / RAID card which can do JBOD.

Yes, I looked at NexentaStor & Open Indiana and I found the performance to be slower than Solaris 11.1. I will see if I can dig the link out with the performance comparisons on it. It also depends on how comfortable you are with cli based interfaces. However, I have been looking at CentOS running QuadStor or Ubuntu running SCST but that's what I am testing at the moment.

 

[RimBlock]

Another option would be to look at Infiniband.

I am curently going this way with EBay sourced parts. Connextx dual port DDR (20GB/s) adapters can be had for GBP 55 (MHGH28-XTC), someone was selling 8mtr CX4 to CX4 DDR cables at US$15 (EBay US). The switch is a bit more tricky but I was lucky to get a good deal from someone on another forum. The one I have is a Flextronics F-X430046 and there is one on EBay (UK) for GBP 350 at the moment.

A 20GB/s one to one link for around GBP 110 + US$15 + shipping is pretty good.

Put a switch in the mix and you have GBP 110 + 2x US$15 + GBP 350 so around GBP 500 for two nodes with around GBP 70-80 per node thereafter.

Software setup is another matter of course.

Note: my current setup is a work in progress so don't take the items above as golden to work together .

I will probably start a thread when all the parts arrive. Last shipment is on the way from the US now.

RB

 

[Shad]

@Andy - Good project log that, thanks for the link. QuadStor looks interesting too, something else for me to try on a prototype.

@RB - Ah yes, Infiinband! If I still had my IB adapters I'd give that a go, but so far it seems that FC is the more established way of doing this and therefore easier to set up. Also, I'm really not sure if I'm going to need or be able to utilise more than 4Gbps, but of course the headroom is always nice.

Keep us posted

 

[khodos]

Hi guys, sorry for the off topic, but I would love to learn the basics about FC and iSCSI. A lot of job opportunities require knowing of SAN technologies, and I know almost nothing about this (I don know the theory about RAID, etc. though).

Any website, webcast, book or similar to start learning in a high level and then start digging?

Thanks

 

[Shad]

Wikipedia is probably as good a place as any for getting a high level summary, followed by some general Google searches on the subject.

On the topic of QuadStor, I found an interesting tidbit in the docs:

Recently support for ZFS ZVOL has been added. Note that QUADStor assumes ownership of about 80% of the total memory. For example on a system with 8GB memory, 6.6 GB of memory shall be reserved by QUADStor for its own use. In order to avoid out of memory conditions the OS and ZFS will have to work with the remaining 1.4 GB.

Those memory usage guidelines may be in place on the assumption that QuadStor's de-dupe and compression are enabled. But even so, it makes me think it would probably be better suited with a hardware RAID config rather than ZFS. Which is a shame.

 

[RSR]

Wikipedia is probably as good a place as any for getting a high level summary, followed by some general Google searches on the subject.

On the topic of QuadStor, I found an interesting tidbit in the docs:



Those memory usage guidelines may be in place on the assumption that QuadStor's de-dupe and compression are enabled. But even so, it makes me think it would probably be better suited with a hardware RAID config rather than ZFS. Which is a shame.

That's one of the reasons I want to get a demo working before moving from Solaris 11.1. I luckly have a hardware RAID card spare, so I would have no problems with testing that.

 

[DRZ]

In terms of relevance to your career? I'd go FC.

FC is easily the better of the two protocols in terms of reliability, meaning that it is the darling of the enterprise. 8G FC isn't any different to iSCSI in bandwidth terms really once you've taken everything into account but iSCSI remains massively less reliable.

Look at the SCSI protocol and you'd see why it is entirely unsuited to transmission over lossy networks - which is why FCoE exists.

FCoE is really a lot nicer than iSCSI as it avoids all of the pitfalls of using iSCSI on a busy network but the downside of needing expensive hardware and retains all of the zoning complexity of FC.

If you can get 4G FC cheaply enough (should be easy) I'd go down that route. You can always set up iSCSI as well and learn the differences (and play with breaking iSCSI by doing not very much...)

 

[Shad]

Cheers DRZ. My hand has been forced by how my 10GbE NICs work under ESXi - basically I can only use one of the two ports without some hacking of the drivers, which I've already tried without success. That means I can't hook the hosts, SAN and my workstation up without a 10GbE switch which is way out of my reach.

But I've got my eye on some 4Gb FC kit on the 'Bay and I've had a play with OpenIndiana and napp-it so I think that's the way I'll go. Well, maybe NexentaStor, but the lack of official FC support in the community edition and the problems people have had after tweaking it into work are a bit off-putting.

 

[DRZ]

At the sort of I/O load you're realistically looking at, iSCSI over a dedicated network is probably going to be more than adequate.

I'd obviously suggest that if you do buy the FC kit you try both so you can see the differences between the two in terms of setup and operation etc etc. Will you be buying enough kit to set up multipathing? That's a pretty key thing to be able to tune etc in VMWare

 

[Shad]

Yeah we're not talking high sustained demand here, but I'd like a decent burst rate. Trouble is I've no idea really what kind of performance the final ZFS config will be capable of.

I've got a test VM at the moment with 6x 50GB vmdks attached and I've got a pool with two disks for parity (the vmdks themselves are on a RAID1 array of Samsung F1 drives). Over iSCSI on my 10GbE adapter I've seen around 125MB/s sustained read and about 30MB/s write, and about 1500 IOPS 4k read. That's seems pretty good for such a basic cut-and-shut prototype - almost too good to be true, given the IOPS are way more than what the underlying hardware is capable of - but I guess that's ZFS and the cache working its magic? The VM only has 2GB of memory too.

Multi-pathing - not something on my radar but the SAN box will be on FC and a 1GbE trunk, so could I look at feeding the hosts the LUN over FC and iSCSI? Not sure there would be much benefit though, other than redundancy.

 

[DiscoDave]

For small/home/test environments it wouldn't really matter which you use, in fact, you should probably use both to get to grips with the different protocols/setups and pitfalls.

If you're looking for a setup to emulate what most enterprises use - go FCoE. I've worked for two large companies and both used FCoE compared to iSCSI. There's a lot to factor in and it's not as simple as 10Gb > 8Gb.

 

[RimBlock]

@Andy - Good project log that, thanks for the link. QuadStor looks interesting too, something else for me to try on a prototype.

@RB - Ah yes, Infiinband! If I still had my IB adapters I'd give that a go, but so far it seems that FC is the more established way of doing this and therefore easier to set up. Also, I'm really not sure if I'm going to need or be able to utilise more than 4Gbps, but of course the headroom is always nice.

Keep us posted

As reqested, a brief update.

The Infiniband cards (MHGH28-XTC - pdf) worked out fo the box for Solaris (no special setup needed). For Windows Server 2012 I needed to flash with the latest firmware, install the drives and then start the Subnet Manager (openSM). Now I can see the whole network (two machine and switch). Flashing the card was a bit of a pain as it was HP branded and the normal Mellanox flash software refused to touch it but another utility they include allows a bypass of various checks and this worked fine. Mellanox have also started a tech community on their site with some very helpful people there. The MHGH28-XTC is a connectx card rather than the older Infinihost cards and seems to be a lot easier to get running.

I just have to setup Solaris to share over SRP (iSER probably) and then I can get to testing. My poolz are fronted by 120GB Agility 3 SSDs with 120GB Intel 520s underprovisioned by 75% as ZIL. These SSDs I had available and I will probably upgrade when cash allows to SSDs with supercaps for the ZIL and maybe Samsung 500GB 940s for the L2ARC.

The cards also support PCoIB and FCoE .

RB

 

[RSR]

Interesting & excellent write-up.

How are you finding the SSD performance with Solaris 11.1? Do you have any bench mark results?

 

[RimBlock]

Well the rejoicing was a little premature.

There are no SRP drivers for Windows 2012 .

So I have;
Windows server (basic domain, backups & media share).
3x ESXi servers (One for home use, two for business testing).
1x Solaris 11.1 (SAN).

The problem is that there is no Subnet Manager for ESXi or Solaris readily available which means I need to run either Windows or Linux on my Infiniband network as both of these have OpenSM available.

After installing WIndows 2012 I find out SRP support has been removed from the drivers so I can run Infiniband but only at 10Gbps not the full 20Gbps DDR speed.

I have now cleared and installed Windows SBS 2011 Essentials and the OpenFabrics drivers do have the SRP miniport driver, only it is refusing to start. From reading around it seems the problem may be when you have more than 2 SRP targets available the SRP miniport driver will refuse to come up on Windows.

It goes without saying I have 4 SRP targets, one for each machine. Luckily VMFS is a multi-access filesystem so I will probably knock it back down to two shares (one for Windows and one for the 3 ESXi boxes) and see if I can get this finally working.

The ESXi boxes can see all 4 SRP shares with the new Mellanox ESXi drivers recently released which now have SRP support.

Solaris seems to be running ok. Fairly different to Linux once you dig under the userspace to an administration level. Setting up the zpools and zfs 'partitions' within the pools was fairly easy though. Comstar management of the Infiniband setup was fine although trying to separate what is needed for SRP only can be a bit confusing as most guides are for iSCSI.

Oh Easter weekend was fun...

Over this week I will sort out having only two targets (or shares on a single target is probably a better description) and see if Windows will work with that then.

RB

 

[Shad]

That's a shame RB, but it sounds like you learned a lot. Albeit the hard way!

The Linux ZFS port was recently declared stable for production use. Maybe you could ditch Solaris and use a Linux distro for your SAN instead?

My 4Gb FC kit should be arriving this week but I'll need to wait a little longer before I can build the SAN.

 

[RimBlock]

Problem is more the Windows side of it if I want to have the 20GBps bandwidth. Solaris seems to be working fine.

I did read today about ZFS and Linux and Infiniband has good support in CentOS (my preferred distro) from what I hear but getting to grips with Solaris is not a bad thing.

My Infiniband card has two ports so I could use one for a Windows Infiniband network with just 1 lun or two available to get round the Windows SRP 2 Lun limit and the other port to the switch for any other non Windows machines. It really feels like a bit of a big kludge to get round driver issues in Windows though.

I could more away from Windows on the server but my home machines are Windows and they all integrate quite well.

Oh well

You FC setup should be rather nice. Those tiny cables are a far cry from the great big thick things I need for Infiniband.

RB

 

[Shad]

A little update from my end then. I have FC kit

4 of these, which although listed as used all appear to be brand new and running the latest QLogic firmware:

One of these, sold as having noisy fans so consequently it went quite cheap:

The SFP modules are all 2Gb but that's fine for getting up and running, and 4Gb modules are cheap enough. The listing was right though, power it up and it buzzed like crazy. So I popped the lid off...

...located the knackered fan and removed it. Since it was next to the PSU I've swapped it with one of the others for now, but will probably replace all 4. Updated the firmware and it's running just fine, nice piece of kit! Connected to my workstation with one of the HBAs in it and that all worked ok too. Next job is to read the manual and learn about zoning.

So now I'm back to working out which SAN distro to use while I wait for another JBOD controller to arrive so I can build the SAN server