4Gb fibre channel vs iSCSI over 10GbE

Nice!

4Gb optics are cheap but make sure you buy ones that are stated working, they are consumables and do break a lot...
 
Nice!

4Gb optics are cheap but make sure you buy ones that are stated working, they are consumables and do break a lot...

Thanks, I bared that in mind and just picked up 4x brand new Finisar 4Gb SFP modules for the princely sum of £7.19 delivered :)
 
So now I'm back to working out which SAN distro to use while I wait for another JBOD controller to arrive so I can build the SAN server :)

I have been meaning to get around to that, as I'd like to get quad store up and running or demo a SCST build on CentOS or Ubuntu.

I'm not 100% sold on the performance of Solaris, CIFS performance to me seems very sub par as well as we'll as the LUN performance.
 
I've decided to go with NexentaStor. It's nice and straight forward for me to get started with and there's a decent level of docs and support out there, and it does VAAI with ZFS.

Anyway I'm sort of up and running now. It's taken a little longer than planned after one of my eBay purchases (SAS 6/iR) turned out to be a dud and needed replacing. So far I've just installed Nexenta and done a basic setup (oh my god, the nightmare of physically getting it on the server - burn ISO to USB? Nope. Mount the ISO over the DRAC? Nope. Find some software to actually burn a shiny disc and dig out a USB CD-ROM drive and wait an eternity for it to install? Yeah...). Took a while to sort the NICs and aggregation out, then set the FC HBA in target mode (found handy guides for all of this though). For some initial testing I've created a 2x2TB pool in a mirror, a 250GB zvol and mapped that to one of my ESXi hosts over the FC switch. Firstly, here's the numbers from the local storage on said host:

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    82.080 MB/s
          Sequential Write :   105.279 MB/s
         Random Read 512KB :    28.259 MB/s
        Random Write 512KB :    32.671 MB/s
    Random Read 4KB (QD=1) :     0.469 MB/s [   114.5 IOPS]
   Random Write 4KB (QD=1) :     0.824 MB/s [   201.2 IOPS]
   Random Read 4KB (QD=32) :     1.325 MB/s [   323.5 IOPS]
  Random Write 4KB (QD=32) :     0.497 MB/s [   121.2 IOPS]

  Test : 1000 MB [C: 36.3% (18.0/49.7 GB)] (x5)
  Date : 2013/04/12 21:35:40
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

Not bad given it's just a RAID1 setup of 1TB Samsung F1 drives. But it starts to grind quickly especially when loading more than one VM at a time (IOPS fail). Here's what Nexenta is giving me right off the bat:

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   344.586 MB/s
          Sequential Write :    79.734 MB/s
         Random Read 512KB :   319.910 MB/s
        Random Write 512KB :    89.099 MB/s
    Random Read 4KB (QD=1) :    32.256 MB/s [  7875.0 IOPS]
   Random Write 4KB (QD=1) :    21.507 MB/s [  5250.7 IOPS]
   Random Read 4KB (QD=32) :   275.660 MB/s [ 67299.7 IOPS]
  Random Write 4KB (QD=32) :    32.078 MB/s [  7831.5 IOPS]

  Test : 1000 MB [F: 1.5% (0.6/40.0 GB)] (x5)
  Date : 2013/04/12 21:27:17
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

Holy crap, that's quite an improvement! Everything but sequential write is a lot better, amazingly so for such a basic disk pool. I'll be adding SSDs for L2ARC and ZIL soon so hopefully those numbers will improve, particularly the write IOPS. The Nexenta box is running dual Xeon 3.0GHz chips, 16GB RAM and 6x 2TB Seagate SATA drives and I'm planning on fitting an Intel 313 20GB drive for ZIL and a-n-other mSATA 128GB SSD for L2ARC.

Really pleased so far though, hopefully more to come from it shortly :)
 
Seems to give a pretty decent turn of speed when the same VMDK is shared from Server 2012 to my Win 7 workstation over 10GbE (normal SMB share):

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   220.104 MB/s
          Sequential Write :    75.649 MB/s
         Random Read 512KB :   178.547 MB/s
        Random Write 512KB :    39.126 MB/s
    Random Read 4KB (QD=1) :    12.100 MB/s [  2954.1 IOPS]
   Random Write 4KB (QD=1) :    10.216 MB/s [  2494.2 IOPS]
   Random Read 4KB (QD=32) :    21.660 MB/s [  5288.0 IOPS]
  Random Write 4KB (QD=32) :    10.481 MB/s [  2558.8 IOPS]

  Test : 1000 MB [F: 1.5% (0.6/40.0 GB)] (x5)
  Date : 2013/04/12 21:57:08
    OS : Windows 7 Ultimate Edition SP1 [6.1 Build 7601] (x64)

:cool:
 
I did wonder that, but I'm not sure what I should be expecting to be honest. Any thoughts on how I'd verify or otherwise test the performance? It just looks like Nexenta/ZFS is doing all the clever read-ahead/cache magic that it's supposed to.

I've created the full 6x2TB RAIDZ pool today and I'm using 3x500GB VMFS zvols with storage vMotion/SDRS doing its magic (very cool). Each VM I move to the SAN is running significantly faster, both in terms of start up time and general usage. I would say the increase in IOPS is there - do I trust the numbers? I don't know... but everything so far feels a lot faster :)

That's a win in my book.

I've set it up multi-pathing with iSCSI too, just in case the FC side of things does a face plant.
 
Last edited:
Interesting results. I see the writes have heavily impacted though looking at a few of the results.

I have just benched marked mine again I am I seeing the following:

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   224.799 MB/s
          Sequential Write :   186.083 MB/s
         Random Read 512KB :   184.480 MB/s
        Random Write 512KB :   188.470 MB/s
    Random Read 4KB (QD=1) :     8.679 MB/s [  2118.9 IOPS]
   Random Write 4KB (QD=1) :     7.547 MB/s [  1842.4 IOPS]
   Random Read 4KB (QD=32) :    95.861 MB/s [ 23403.7 IOPS]
  Random Write 4KB (QD=32) :    66.651 MB/s [ 16272.1 IOPS]

  Test : 1000 MB [C: 44.5% (22.1/49.7 GB)] (x5)
  Date : 2013/04/14 15:41:00
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   230.229 MB/s
          Sequential Write :   206.861 MB/s
         Random Read 512KB :   184.449 MB/s
        Random Write 512KB :   192.574 MB/s
    Random Read 4KB (QD=1) :     8.360 MB/s [  2041.1 IOPS]
   Random Write 4KB (QD=1) :     7.189 MB/s [  1755.0 IOPS]
   Random Read 4KB (QD=32) :    96.432 MB/s [ 23542.9 IOPS]
  Random Write 4KB (QD=32) :    92.672 MB/s [ 22624.9 IOPS]

  Test : 1000 MB [C: 37.8% (37.6/99.7 GB)] (x5)
  Date : 2013/04/14 15:43:20
    OS : Windows NT 6.2 Enterprise Edition [6.2 Build 9200] (x64)

I have 4 x Intel 335 240GB striped over two ZFS volumes, which presents via comstar FC. Then I have a sDRS setup which should balance the load between these two LUNs. I do have around 25 VM which are active on these volumes which isn't too bad I guess.

However, I do think is poor performance for SSD over FC. I am getting the impression Solaris 11.1 doesn't play well with SSDs
 
Yes I'd have thought it would run faster than that on the SSDs you have.

I'm running into a bit of an issue now - copying a couple of TB of mixed files (some small, some very big) over 10GbE onto a 3TB LUN mounted as RDM on a VM, I'm finding it's causing the other VMs to bog down. A couple have even been reset by HA after not responding. I've been adjusting the disk resource allocations to lower the priority of the file servers I'm working on and raise the priority on a couple of the other VMs, which I think is helping.

Am I likely to see an improvement in this area by using a 20GB SSD for ZIL?
 
Those are good results. But i would have thought it could go higher. At most to 500mbyte which is 6gbps sata limitation. But considering it is in raid configuration it theoretically should be able to go more than 6gbit to the 10gbit max from the interface. Otherwise it is not realy utilizing the 10gigabit. I would be interested to see a comparison with 4gigabit iscsi or fc using same disks etc.
 
Last edited:
Am I likely to see an improvement in this area by using a 20GB SSD for ZIL?

I did have add a spare SSD I have to add the ZIL. In testing for me it made little or no difference.

Yes I'd have thought it would run faster than that on the SSDs you have.

This is what's lead me on to look at other solutions and some what confused. :(

I'm running into a bit of an issue now - copying a couple of TB of mixed files (some small, some very big) over 10GbE onto a 3TB LUN mounted as RDM on a VM, I'm finding it's causing the other VMs to bog down. A couple have even been reset by HA after not responding. I've been adjusting the disk resource allocations to lower the priority of the file servers I'm working on and raise the priority on a couple of the other VMs, which I think is helping.

What block size (512b or 4KB) and record size (128K) are you running? I know Nexenstor has a lot of back ported updates to do with this. Also if you do a zfs get all <vol_name> whats does this show?
 
Those are good results. But i would have thought it could go higher.

If you mean my results they really aren't that great.

But i would have thought it could go higher. At most to 500mbyte which is 6gbps sata limitation. But considering it is in raid configuration it theoretically should be able to go more than 6gbit to the 10gbit max from the interface. Otherwise it is not realy utilizing the 10gigabit. I would be interested to see a comparison with 4gigabit iscsi or fc using same disks etc.

It completely depends on the media that's being used to store the information, as SATA drive is still unable to fully saturate a SATA II (3Gb) connection. Also when talking about storage IO, you often present one or more LUNs from difference aggregate (logical grouping of discs). So no a single disk LUN maybe not able to fully saturate the 10Gb link but a number of them (LUNs) will be.

4Gb iSCSI will be slower than a 4Gb FC, due to TCP/IP overheads. As you will often find that 10Gb iSCSI will perform the same as 8GB FC. This is the difference between a proper storage protocol Vs one that's just piggy backs on a TCP/IP packet. Plus you are also going to be reliant on MPIO to give you the extra bandwidth.
 
Code:
root@fcsan1:/volumes# zfs get all FCSAN1_Vol1
NAME         PROPERTY              VALUE                  SOURCE
FCSAN1_Vol1  type                  filesystem             -
FCSAN1_Vol1  creation              Sat Apr 13 12:43 2013  -
FCSAN1_Vol1  used                  7.83T                  -
FCSAN1_Vol1  available             1.05T                  -
FCSAN1_Vol1  referenced            162K                   -
FCSAN1_Vol1  compressratio         1.14x                  -
FCSAN1_Vol1  mounted               yes                    -
FCSAN1_Vol1  quota                 none                   default
FCSAN1_Vol1  reservation           none                   default
FCSAN1_Vol1  recordsize            128K                   default
FCSAN1_Vol1  mountpoint            /volumes/FCSAN1_Vol1   local
FCSAN1_Vol1  sharenfs              off                    default
FCSAN1_Vol1  checksum              on                     default
FCSAN1_Vol1  compression           on                     local
FCSAN1_Vol1  atime                 on                     default
FCSAN1_Vol1  devices               on                     default
FCSAN1_Vol1  exec                  on                     default
FCSAN1_Vol1  setuid                on                     default
FCSAN1_Vol1  readonly              off                    default
FCSAN1_Vol1  zoned                 off                    default
FCSAN1_Vol1  snapdir               hidden                 default
FCSAN1_Vol1  aclmode               discard                default
FCSAN1_Vol1  aclinherit            restricted             default
FCSAN1_Vol1  canmount              on                     default
FCSAN1_Vol1  xattr                 on                     default
FCSAN1_Vol1  copies                1                      default
FCSAN1_Vol1  version               5                      -
FCSAN1_Vol1  utf8only              off                    -
FCSAN1_Vol1  normalization         none                   -
FCSAN1_Vol1  casesensitivity       sensitive              -
FCSAN1_Vol1  vscan                 off                    default
FCSAN1_Vol1  nbmand                off                    default
FCSAN1_Vol1  sharesmb              off                    default
FCSAN1_Vol1  refquota              none                   default
FCSAN1_Vol1  refreservation        none                   default
FCSAN1_Vol1  primarycache          all                    default
FCSAN1_Vol1  secondarycache        all                    default
FCSAN1_Vol1  usedbysnapshots       0                      -
FCSAN1_Vol1  usedbydataset         162K                   -
FCSAN1_Vol1  usedbychildren        7.83T                  -
FCSAN1_Vol1  usedbyrefreservation  0                      -
FCSAN1_Vol1  logbias               latency                default
FCSAN1_Vol1  dedup                 off                    default
FCSAN1_Vol1  mlslabel              none                   default
FCSAN1_Vol1  sync                  standard               default
The RDM mounted zvol I'm writing to at the moment is using 4KB blocks to match up with the NTFS partition on it. The VMFS zvols are all set to 8KB which was the default.
 
I few things id recommend / test are:


checksum=off
atime=off
casesensitivity=mixed (if you have CIFs on the same volume)
logbias=throughput (This is your logging (ZIL), if you change it to throughput doesn't use logging)
primarycache - ARC
secondarycache - L2ARC I would also recommend / test changing this to metadata.

I also notice you have compression on, what algorithm is it using? LZJB?

These are a few useful list link:

http://icesquare.com/wordpress/how-to-improve-zfs-performance/
 
Thanks Andy - I've read up on those settings and they seem good, although I can't work if atime only applies CIFS/NFS storage on Nexenta and not to SCSI LUNS?

Anyway, I think I've broken it somehow. Currently getting abysmal performance, 15MB/s seq read and about 100 IOPS. All zvol and pool settings are back to how they were. I'm migrating my VMs back to local storage and I'll give it a reboot, hopefully fixing it...!
 
After a reboot and migrating one of my VMs back I'm getting some far more realistic numbers now:

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   292.939 MB/s
          Sequential Write :    83.412 MB/s
         Random Read 512KB :   257.021 MB/s
        Random Write 512KB :   100.759 MB/s
    Random Read 4KB (QD=1) :    14.634 MB/s [  3572.7 IOPS]
   Random Write 4KB (QD=1) :    11.121 MB/s [  2715.0 IOPS]
   Random Read 4KB (QD=32) :    22.055 MB/s [  5384.6 IOPS]
  Random Write 4KB (QD=32) :    15.636 MB/s [  3817.4 IOPS]

  Test : 1000 MB [C: 38.2% (19.0/49.7 GB)] (x5)
  Date : 2013/04/15 18:06:31
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

Bit of a worry though. I don't know enough about ZFS or Nexenta to know what my reboot fixed but I'll carry on loading up with the rest of my VMs and keep an eye on the figures... :confused:
 
Starting to get somewhere with this little project now. NexentaStor has gone though. The issue I was having above was consistently reproducible by copying around 1TB of mixed data onto a LUN via a VM. The ARC would stop being used completely and the performance (for all LUNs) just went through the floor. The only way I could fix it was with a reboot.

So now I'm running OmniOS and Napp-It. I hadn't heard of OmniOS before but it's recommended for Napp-It and seems nice and light weight. Now I've learnt my way around the Napp-It menus I'm quite enjoying it. Best of all though the performance is excellent - no ARC problems after the same file copying. I've changed a couple of things though - am now running the following ZFS pool instead of RAIDZ:

Code:
FCSAN1_Vol1
  mirror-0
    c13t0d0
    c13t1d0
  mirror-1
    c13t2d0
    c13t3d0
  mirror-2
    c13t4d0
    c13t5d0
Basically RAID10. I also had some funny issues with ESXi when using RDM for some large LUNs, in that after a while the VM would seem to have intermittent connection with the drives. So I've changed to having a 4.2TB VMFS datastore with a couple of 1.5TB VMDKs configured as a dynamic drive in Windows Server (this is in addition to 3x 300GB VMFS datastores for virtual machines). This drive is one half of a DFS replica so I'm not concerned about the negatives of dynamic drives or striping the two volumes. Bottom line is it hasn't broken yet as it was doing before :)

I'll sort out some benchmarks once I've finished moving things around.
 
Excellent. I look forward to the bench marks.

I am just about to go back to openfiler to review and bench mark.

Just run a bench in W2012 to get a good benchmark figure. :o:cool:

SSD Volume

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : [url]http://crystalmark.info/[/url]
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :  1476.637 MB/s
          Sequential Write :  1030.035 MB/s
         Random Read 512KB :  1172.646 MB/s
        Random Write 512KB :  1146.336 MB/s
    Random Read 4KB (QD=1) :    57.557 MB/s [ 14051.9 IOPS]
   Random Write 4KB (QD=1) :    83.092 MB/s [ 20286.2 IOPS]
   Random Read 4KB (QD=32) :   490.447 MB/s [119738.0 IOPS]
  Random Write 4KB (QD=32) :   197.807 MB/s [ 48292.8 IOPS]

  Test : 1000 MB [G: 0.0% (0.2/892.1 GB)] (x5)
  Date : 2013/04/20 18:55:07
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

RAID 5 Volume

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   900.838 MB/s
          Sequential Write :   508.524 MB/s
         Random Read 512KB :   162.570 MB/s
        Random Write 512KB :   146.932 MB/s
    Random Read 4KB (QD=1) :     1.499 MB/s [   366.0 IOPS]
   Random Write 4KB (QD=1) :     2.064 MB/s [   503.9 IOPS]
   Random Read 4KB (QD=32) :     8.557 MB/s [  2089.1 IOPS]
  Random Write 4KB (QD=32) :     2.631 MB/s [   642.2 IOPS]

  Test : 1000 MB [F: 0.0% (0.4/7449.9 GB)] (x5)
  Date : 2013/04/20 19:04:04
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

I have just been testing Datacore SAN Symphony-V and the results look pretty good so far. Using a basic setup with the LUN presented as a RDM. All other settings are completely default.

Code:
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   294.316 MB/s
          Sequential Write :   346.895 MB/s
         Random Read 512KB :   286.544 MB/s
        Random Write 512KB :   325.697 MB/s
    Random Read 4KB (QD=1) :    37.293 MB/s [  9104.8 IOPS]
   Random Write 4KB (QD=1) :    13.491 MB/s [  3293.8 IOPS]
   Random Read 4KB (QD=32) :   210.328 MB/s [ 51349.6 IOPS]
  Random Write 4KB (QD=32) :    73.196 MB/s [ 17870.0 IOPS]

  Test : 4000 MB [E: 0.0% (0.2/892.1 GB)] (x5)
  Date : 2013/04/21 8:43:08
    OS : Windows NT 6.2 Server Standard Edition (full installation) [6.2 Build 9200] (x64)
 
Last edited:
Sorry to take this thread slightly off topic but I have a couple of questions!

Firstly I'm just mucking around with this (no business of vital data etc) so happy to play around. I have a ESXI server and I've wanted to build a Fibre SAN, just because I can.

Building the SAN was fine, Running Solaris 11 and COMSTAR. Hardware wise I have passed through my Emulex 4GB dual port HBA and LSI 9260-8i with 4x 1TB WD Blacks in RAID 0. Now I know this RAID 0 array will do about 500MB read and 400MB write when connected to windows however when I connect it through fibre channel to my main windows PC (4GB qLogic HBA) this drops to 50MB read/write.

Can anyone offer any advise on how to speed this up a bit? (Can you tell I don't really know what I'm doing?) Anyway any help would be greatly appreciated!
 
Back
Top Bottom