Trials and tribulations of a new Admin.

What OS is the VM host?

What are the switch port settings set to?

Collisions & errors can point to duplex mismatch, faulty cables and a number of other things, including incorrect teaming settings in the OS and on the switch.

OSs are Server 2012 R2 and Hyper-V Server 2012 R2. Both those are set to Dynamic LACP and the switch is set to LACP for the LAG.

Haven't had any time to do any troubleshooting yet as other stuff has been higher priority, and whilst it's dropping packets, it's not actually breaking anything.

Having said that, just had to head in as I couldn't get on. The main switch decided to flunk out. And as everything runs through it I didn't get any notifications. So there's another reminder that 1) there's no out of band management - everything is on the same network and subnet etc. and 2) there's no redundancy on the main switch.

Fortunately it just needed a restart.

just throwing some ideas on this packet loss

is IPV6 enabled - if so disable it using this command

Set-Net6to4Configuration -State Disabled


And ISATAP

Set-NetIsatapConfiguration -State Disabled


also worth disabling any Offloading configuration on the network card settings

As I understand it MS strongly reccommend not to disable IPv6?

I think I've narrowed it down to the hyper-v host with inbound packets. Checking the firmware it showed up to date, but checking the part number for the NIC itself showed an update so I'm just waiting for a free moment to install that. If that doesn't work I'll try disabling features.

Thanks

Test disabling "Virtual Machine Queues" on the hosts NIC's.
That can cause some issues as well.

GDL said:

Test disabling "Virtual Machine Queues" on the hosts NIC's.
That can cause some issues as well.

Click to expand...

I've seen this cause big headaches with Broadcom based NICs. IIRC, newer firmware helps but I think we left it turned off.

Most things seem to cause headaches with Broadcom NICs.

The particular NICs are Broadcom but don't have VM traffic on them, they're reserved for backup traffic.

nutcase_1uk said:

The particular NICs are Broadcom but don't have VM traffic on them, they're reserved for backup traffic.

Click to expand...

Where was it you were seeing the dropped packets?

During the backup routine. Veeam copies from the Hyper-v server to the repository and the veeam repository to the tape library (which is connected to the hyper-v server).

nutcase_1uk said:

During the backup routine. Veeam copies from the Hyper-v server to the repository and the veeam repository to the tape library (which is connected to the hyper-v server).

Click to expand...

Yeah, no, but where are you seeing? On the switch?

Ah, apologies. On the switch. I've got a free solarwinds trial going:

Blue receive, green transmit.

Lag switch to hyper-v host

http://imgur.com/ZDSTYIY

http://imgur.com/fsGPMzi

Lag switch to veeam repo:

http://imgur.com/0KiM4qD

http://imgur.com/AcsQ7k6

So packets from the switch to the hosts are being discarded - so not the switch? All the suggested changes have been made to the veeam host but still not had a window to change the hyper-v host.

nutcase_1uk said:

<snipped some graphs>

So packets from the switch to the hosts are being discarded - so not the switch? All the suggested changes have been made to the veeam host but still not had a window to change the hyper-v host.

Click to expand...

Brilliant. You can see from the utilisation graphs that you are only using one GbE worth in each direction -- 50% for the 2x1GbE link, and 33% for the 3x1GbE link. Which is consistent with how LACP works anyway, it assigns one path to one flow, and it's only when you have multiple flows that you start seeing the benefits.

Until you have a chance to understand a bit better how the LACP configuration is interacting between the switch and the servers, I would get rid of it.

What model of switch is it?

Yeah it has always been doing that (also mentioned above somewhere). I've taken it off of the veeam repo, will see what that does during tomorrows run.

Switch is one of netgear's finest - GS748Tv5

No apparent difference - still transmit packet loss from switch to veeam.

nutcase_1uk said:

No apparent difference - still transmit packet loss from switch to veeam.

Click to expand...

Wait, so you completely got rid of the teaming and you are still seeing massive packet loss?

What do the error counters on the switch port show? If your switch can't give you this information then throw it away.

rotor said:

Wait, so you completely got rid of the teaming and you are still seeing massive packet loss?

Click to expand...

On one end (veeam to switch), have now removed it from the other end.

A screenshot I have before I cleared the lags off, shows transmit packets discarded 14736585

Now that google have killed off picasa, anyone know of an easy upload site?

imgur.

Havent caught up on the whole thread, but one thing causing us some headaches lately was someone turned on Jumbo frames on some new kit when the switches weren't configured for them. That led to random packet losses for us - when a packet went over a particular size, or the amount of packets ramped up significantly, our switches would go ga-ga.