UNRAID network issues

[mikeyosm]

Hi

 

I need some help troubleshooting a niggling network issue with UNRAID 6.2.x and 6.3.0 RC releases.

My UNRAID host is connected to a HomePlug 1Gbps adaptor (i have both Solwise and Devolo).

I have set up 1 network bridge for my VMs with an MTU of 9000 and all my OS NICs are set to jumbo frames (9000).

On first boot up, I have absolutely no issues connecting to the local LAN or the Internet, all speeds are as expected.

 

However, on completely random occasions (but mainly and more noticeably whilst online gaming), the network connection drops! It only drops for about 1-3mins and then restores itself.

When this happens, I try to isolate whether it's a VM, host or HomePlug issue. So, when the connection drops, I ping the router's default gateway from two different VMs and UNRAID console. Both VMs and the console fail PING. I then switch from Devolo HomePlug to Solwise HomePlug and the same connection issue occurs so I know it's not the HomePlug adaptors. This has been happening for many months now and in this time I have replaced my motherboard also.

 

I am at a loss as to why I get an intermittent network disconnects.

 

Is there any diagnostics that I can run in the background in UNRAID so that I can capture the state of the NIC before and after the issue occurs?

 

My NICs: (tried both same result)

53:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)

00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1] (rev 05)

 

Network Settings:

Static IP

MTU 9000

Google DNS servers

Bonding - No

 

VM Network on both VMs:

  <interface type='bridge'>

      <mac address='xx:xx:xx:xx:xx:xx'/>

      <source bridge='br0'/>

      <target dev='vnet0'/>

      <model type='virtio'/>

      <alias name='net0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

[Zonediver]

Disable the Jumbo-Frames - thats the point 

[mikeyosm]

Disable the Jumbo-Frames - thats the point 

If I do that, I won't get transfer speeds above 100MB/s between my UNRAID host SMB shares and my VMs. I have SSDs for everything and want to get (and do get) 300MB/s transfer speeds with Jumbo frames enabled and an MTU of 9000 on my VM virtual NICs

[Zonediver]

I dont use jumbos and get 99-115MB/s over the LAN - depending on the NIC-interface (PCIe) - but i dont use VMs.

I remember there are some issues to use jumboframes but its possible that they are fixed in the latest unraid release - just a guess.

[mikeyosm]

OK, so let me get this straight. I thought enabling JF to 9000 allowed me to *theoretically* transfer files up to 10Gb/s speeds? My VNICs are set to 10GBps on my VMs.

If I don't set MTU 9000, I am limited to approx 100-120MB/s transfer speeds?

[Zonediver]

10Gb/s ? 

Your Adapters are 1000MBit - not 10GBit 

They can transfer max. 1GBit or near 110-115MB/sek

The jumboframe is only for transfering larger blocks of data - not mor date on the NIC - more than 1000MBit is not possible with this adapters.

[mikeyosm]

The physical adaptors are 1Gbps - my Virtual NICs inside my W10 VM are 10Gbps! If the MTU is set on UNRAID to 9000, this lets me use the VNICs @ 10Gbps to their full potential. Setting the MTU back to default would not help me achieve speeds above 120MB/s!

[Zonediver]

I am confused now...

The limit on your Intel-NICs is 1000MBit - dosnt matter if your virtual VM-NIC is set to 10GBit...

[mikeyosm]

Ill try to make this clear.

 

I set 9000 MTU on UNRAID eth0. Doing this creates a bridge interface (br0) used for the VMs with an MTU of 9000.

You cannot set an MTU on br0 manually, it needs to be set on the physical interface, in this case my Intel 1Gbps NIC.

Now that the bridge used for my VMs is at 9000 MTU, I can set the 10Gbps VNICs to 9000 MTU. This lets me transfer files between UNRAID host and VMs at speeds greater than 300MB/s.

I know this works because I do it all the time.

[ken-ji]

Ill try to make this clear.

 

I set 9000 MTU on UNRAID eth0. Doing this creates a bridge interface (br0) used for the VMs with an MTU of 9000.

You cannot set an MTU on br0 manually, it needs to be set on the physical interface, in this case my Intel 1Gbps NIC.

Now that the bridge used for my VMs is at 9000 MTU, I can set the 10Gbps VNICs to 9000 MTU. This lets me transfer files between UNRAID host and VMs at speeds greater than 300MB/s.

I know this works because I do it all the time.

 

I can only guess but I'm fairly sure the Home plug adapters were never designed for jumbo frames, thus when you start trying to transfer bulk data between unraid/vms to your local network/internet, the connection can drop until the system resyncs/recovers.

 

In your particular case, I'm going to recommend to setup an additional vnic on the VMs and connect to unraid via the original vibr0 bridge

[caveat: I'm not sure if a custom bridge is better]

 

that way they can talk to unraid using the 2nd virtual nic with jumbo frames, and still talk to the rest of the LAN/internet using regular frames.

 

[mikeyosm]

Ill try to make this clear.

 

I set 9000 MTU on UNRAID eth0. Doing this creates a bridge interface (br0) used for the VMs with an MTU of 9000.

You cannot set an MTU on br0 manually, it needs to be set on the physical interface, in this case my Intel 1Gbps NIC.

Now that the bridge used for my VMs is at 9000 MTU, I can set the 10Gbps VNICs to 9000 MTU. This lets me transfer files between UNRAID host and VMs at speeds greater than 300MB/s.

I know this works because I do it all the time.

 

I can only guess but I'm fairly sure the Home plug adapters were never designed for jumbo frames, thus when you start trying to transfer bulk data between unraid/vms to your local network/internet, the connection can drop until the system resyncs/recovers.

 

In your particular case, I'm going to recommend to setup an additional vnic on the VMs and connect to unraid via the original vibr0 bridge

[caveat: I'm not sure if a custom bridge is better]

 

that way they can talk to unraid using the 2nd virtual nic with jumbo frames, and still talk to the rest of the LAN/internet using regular frames.

 

Interesting idea. What is the difference between br0 and virbr0 virtual interfaces? How would I create a new bridge, i only get an option in settings/vm manager to choose either of those two bridges. Please explain in a bit more detail what I need to do.

 

thanks

 

[Bean]

very new here and haven't gotten around to tinker with unRAID myself but I did stumble over this part in the documentation. Maybe it's of some help to you.

 

http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management#Configure_a_Network_Bridge

 

Good luck

[ken-ji]

Ill try to make this clear.

 

I set 9000 MTU on UNRAID eth0. Doing this creates a bridge interface (br0) used for the VMs with an MTU of 9000.

You cannot set an MTU on br0 manually, it needs to be set on the physical interface, in this case my Intel 1Gbps NIC.

Now that the bridge used for my VMs is at 9000 MTU, I can set the 10Gbps VNICs to 9000 MTU. This lets me transfer files between UNRAID host and VMs at speeds greater than 300MB/s.

I know this works because I do it all the time.

 

I can only guess but I'm fairly sure the Home plug adapters were never designed for jumbo frames, thus when you start trying to transfer bulk data between unraid/vms to your local network/internet, the connection can drop until the system resyncs/recovers.

 

In your particular case, I'm going to recommend to setup an additional vnic on the VMs and connect to unraid via the original vibr0 bridge

[caveat: I'm not sure if a custom bridge is better]

 

that way they can talk to unraid using the 2nd virtual nic with jumbo frames, and still talk to the rest of the LAN/internet using regular frames.

 

Interesting idea. What is the difference between br0 and virbr0 virtual interfaces? How would I create a new bridge, i only get an option in settings/vm manager to choose either of those two bridges. Please explain in a bit more detail what I need to do.

 

thanks

 

br0 is a "physical bridge". eth0 (or bond0 - the bonding of all your eth ports) is part of this bridge and allows traffic to go out to the LAN.

virbr0 is a "logical bridge".there are no physical connections to the LAN, but unRAID and any VMs can talk over this lan securely (its all software, but the OS network stack is still involved.)

Sorry about the confusion regarding custom bridges - please ignore that as there should be issue there.

 

Now since br0 is connected to the LAN, using MTU 9000 will cause all attempts to talk to the LAN to have issues if you are sending large packets, because unraid (or the VMs) will not fragment the packets until they exceed 9000bytes (MTU) but other LAN devices including the router will barf at the malformed packets (too big - MTU is only 1500). this is what's causing your issues (I think)

 

[miniwalks]

Just disable jumbo frames.

Even using 10gbps and 40gbps NICs there is little difference in "real world" throughputs.

 

1gbps NIC, you'll see 115mbyte/sec (125max - overheads)

10gbps NIC, 1.125 gbyte/sec

 

Jumbo frames are a bit of a misnomer unless you 300% know what you're doing, what the workload is, and what the hardware/software in the path is

 

 

Sent from my iPhone using Tapatalk