VM storage performance? Perfmon query.

Lanz

Lanz

Lanz

Soldato
Joined
26 Nov 2002
Posts
6,852
Location
Romford
Hi

I have an Win2003 Server running Virtual Server 2005R2. Its running just 10 VM's doing various activities. I want to migrate these VMs to a new Hyper-V server that's connected to 1Gb iscsi Sun NAS.

Will I have trouble running lots of VMs from a 1gig channel? I'm used to using 4GB FC connections, so ethernet iscsi is new to me.

When I look at the perfmon disk counter for my current host server, the 10 VMs are using an about 30,000 Avg. Disk Bytes/Transfer, peaking upto about 60,000. This seems quite low to me, am I looking at the correct counter?

Any advice or tips would be great.

cheers
 
Forgive me, but I'm unfamilar with ESX and diagrams like this.

Storage1 = your Sun SAN?

And the `System` Data` `Citrix` etc are Shares/LUNs?
 
What roles are your VMs doing?

Mail is a big disk hit.... something else to think about is also swapping - any time a VM needs to swap it's going to thrash the disk heavily. Disk I/O is expensive, but RAM is so cheap these days you'll want to go for as much as you can.

1gbit is about 100MB/s once overheads are considered etc - consider you'll get about 30-50MB/s from a single SATA disk on average, you're really not giving your VMs much... does the Sun have multiple ports? You might want to think about adding a 4 x 1Gbit NIC and then running multiple connections to your front end machine(s).
 
Last edited:
The VMs are mostly app servers, running java/iis/apache type systems. We wont be virtualising any mail, database or file servers anytime soon. And if we do, these will be connected to our better Dothill SAN via FC.

The Sun Storage has 4 x Gbit connectors on the back, we are currently aggrogating 2 of them together (leaving 1 for management and 1 for intersite replication), giving a 2GB connection from the switch, but the actual host servers will probably only connect to the switch via 1 Gbit interface. So for the time being 100MB/s is probably all we can manage.
 
Last edited:
I think you'll be OK with that throughput, it's the actual I/O rather than the throughput that will hammer a SAN/NAS in my experience (especially in most VMs usage). The only thing is that if someone is extracting a big zip file on a VM, or moving loads of big files etc, you don't want the other VMs on the node to crawl to a halt.
 
Lanz said:
Forgive me, but I'm unfamilar with ESX and diagrams like this.

Storage1 = your Sun SAN?

And the `System` Data` `Citrix` etc are Shares/LUNs?
Click to expand...

Indeed - they are different LUNs which correlate to different arrays. All through the same iSCSI controller though
 
daz said:
I think you'll be OK with that throughput, it's the actual I/O rather than the throughput that will hammer a SAN/NAS in my experience (especially in most VMs usage). The only thing is that if someone is extracting a big zip file on a VM, or moving loads of big files etc, you don't want the other VMs on the node to crawl to a halt.
Click to expand...

I dont think there's an option for throttling disk throughput like you can throttle CPU % per VM.

So how to others setup their VMware/Hyper-V guests with iscsi so this doesnt happen? Do you have multiple LUNs, and only put 2 or 3 guests on each interface. I'm gonna quickly run out of ports if that's the case, as I have multiple Virtual networks already to share the load.

Or is the OS intelligent enough to dymanically assign resources so 1 VM doing big IO doesnt effect the others, and they just share it.?
 
I don't think there are any ways of disk scheduling per VM on this type of system at the moment - this is why disk intensive applications don't go down very well on Amazon EC2 and other "cloud" based systems etc.

You'll probably be fine, but as said, make sure you give each VM as much RAM as possible to avoid swapping.
 
I take it, restarts, patching and disk intensive things like that should just be shedualed so that only one VM is doing that kind of work at anyone time. Things like setting the start time for each VM for 2 mins apart when the host server reboots.

This is all new to me, as all our current VMs are running from local SAS disks, so I've never had to really think about these things before. But we have bought about 48TB of Sun Storage and need to do something with it....
 
I did a quick 2 hour perfmon test of our current VM host and it's maximum throughput was about 18MB/s for the 10 VMs. I'll do a new log tonight, for 24hrs which also measures Split IO.
 
Look at Writes/Reads per sec average/peak.

Also can you let me know the disk layout behind the datastore(s).

It will undoubtedly be fine as don't forget that there's always cache to hit even though you'll essentially randomise the data stream.

Also, as you said iSCSI NAS, is it fake iSCSI, ie block level protocol on a virtualised FS? There will be a bit of overhead on this.

Also, 48TB? Is this all NAS space?
 
Lanz said:
I dont think there's an option for throttling disk throughput like you can throttle CPU % per VM.

So how to others setup their VMware/Hyper-V guests with iscsi so this doesnt happen? Do you have multiple LUNs, and only put 2 or 3 guests on each interface. I'm gonna quickly run out of ports if that's the case, as I have multiple Virtual networks already to share the load.

Or is the OS intelligent enough to dymanically assign resources so 1 VM doing big IO doesnt effect the others, and they just share it.?
Click to expand...

Correct, you can't schedule disk, you should plan the disk layout so its not a problem. To clarify mine (above)

System - 15k SAS RAID10 for system drives and transaction logs
Data - 15k SAS RAID5 for database files
Citrix - 15k SAS RAID5 for Citrix Servers (separating intensive IO machines)
SATA - 7.2k SATA RAID5 for machines that don't require high IO or require large capacity

I've set up one LUN per array, but (depending on your SAN) you could set up multiple LUNs on the same array and cap throughput on the SAN. Nothing stopping you having an Exchange LUN, an SQL LUN etc etc.

I've had no problems with the fabric at all - it's set up with 2 NICs on each host for resilience, one on board and one from the addin card. They are connected to 2 HP Procurve GB switches which are both connected to the 2 iSCSI controllers - 4 paths total.

You're more likely to hit the limit of the array before you max out the fabric. Someone once told me you need about 48 disks before you need FC (roughly!)

Lanz said:
I take it, restarts, patching and disk intensive things like that should just be shedualed so that only one VM is doing that kind of work at anyone time. Things like setting the start time for each VM for 2 mins apart when the host server reboots.

This is all new to me, as all our current VMs are running from local SAS disks, so I've never had to really think about these things before. But we have bought about 48TB of Sun Storage and need to do something with it....
Click to expand...

Its not very often you'd be doing restarts and patching during the day anyway - never had even a hint of a problem with throughput even at the weekends when patching, reboots, virus scans VM backups and guest level backups are occurring all at the same time.

What storage array are you using out of interest?
 
How many disks have you got? The J4400 takes up to 24x 1TB so have you got 2 of them? As 48 spindles is pretty high throughput.. Sun quote 2GBytes/sec network throughput for their 48 disk thumper. The 10 VMs you're planning to run will be fine on 1Gbit but your fabric will bottleneck before the array
 
It depends how the array is being presented to the VM node though. I imagine you're running ZFS somewhere - do you have any flash/NV RAM acceleration in the 7310? Have you played with any settings?
 
Yeah 2 of them. They are new, nothing yet on them, we have configured both sets of 24 as one big pool, I think its Raid6+1, as that's what they recommended for a good balance of performance + redundacy.

We will definately be limitied by the 4 x 1GB ports, but the switch we bought has a couple of 10Gbe ports, so we can do a quick upgrade on the back of the NAS if needs be.

It has a couple of read SSDs for buffering and 16GB of RAM. Not played that much with it, as we are still awaiting the proper switches to be delivered.
 
Yeah 1 big 48 disk RaidZ2 isn't going to do so great for VMs. Do some benchmarks and workout whats latency sensitive and what needs high throughput and create multiple mirrored pools as required. You can set the SSDs as a striped logzilla/readzilla or 1 for each.. again depends how much ZIL and L2ARC get hammered as to what is best. Also if you're using COMSTAR for iSCSI there's lots of tuneables that will help!
 
Its set for `Double parity RAID` I didnt see any mention of Raid Z or Z2 etc when configuring.
 
You must log in or register to reply here.
Back
Top Bottom