Need your help with Intel I350-T4 NIC (Quad port - pass-through individual ports)


Recommended Posts

Hello,

 

I have been using unRaid for the past year half.  Its been the best thing ever!  THANK YOU LIME Technology and others!

So I am needing some help now.  I know that this has been hashed bashed and trashed over a bit.  I have done all my searching.  I have tried a number of different findings and still no luck with this. 

 

Here is my current setup

 

System Information

Model: Custom
M/B: Supermicro - X10SL7-F
CPU: Intel® Xeon® CPU E3-1241 v3 @ 3.50GHz
HVM: Enabled
IOMMU: Enabled
Cache: 256 kB, 1024 kB, 8192 kB
Memory: 32 GB (max. installable capacity 32 GB)
Network: eth0: 1000 Mb/s, full duplex, mtu 1500 
 eth1: not connected
Kernel: Linux 4.9.10-unRAID x86_64
OpenSSL: 1.0.2k
Uptime: 

 

PCI Devices and IOMMU Groups

IOMMU group 11
	[8086:1533] 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 12
	[8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 13
	[8086:1521] 07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
IOMMU group 14
	[8086:1521] 07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
IOMMU group 15
	[8086:1521] 07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
IOMMU group 16
	[8086:1521] 07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection

Here is what I am trying to accomplish:

I want to pass-through the last two individual ports of my Intel I350-T4 NIC to one of my VM that is running Debian.  So that would be [8086:1521] 07:00.2 and [8086:1521] 07:00.3 that I want to use on that Quad port NIC.  BUT from my reading and self education on this, I cannot pick out these ports separately and leave the others for unRaid.  And that is fine as I would like to use the other two for two other VMs.  So I would need to stub the whole card so that unRaid doesn't see it and want to use it.

 

Here is what I have done so far:

 

I have added this to the syslinux.cfg:

append pci-stub.ids=8086:1521 initrd=/bzroot

I have even tried 

append vfio-pci.ids=8086:1521 initrd=/bzroot

 

I personally do not know what the difference is.  But they both seem to do the same thing and allowing the WHOLE NIC card to be stubbed from the unRaid server.

After a reboot of the unRaid server.  I see that this is successful in this part of the steps.

 

Now onto the addition of the two ports that I want a Debian VM to use.  Here is what I can see and then what two ports I have selected.

 

58b1d457c906d_vmedit.thumb.jpg.daefa37e0cf99b9d35e5be2056bc21b0.jpg

 

And here is the XML for this VM image

 

PBX xml.txt

Here is the most important part of the whole thing.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </hostdev>

All of this should be right from all the research that I have done. But the VM does not see the two cards.

 

Here are all the Hidden different posts that I have found outside of the sticky main posts on this subject.  These sure did help me get to this point!  and I want to acknowledge the person behind each one and say Thank you!

I feel that there are more posts that I have read on this subject.  Its not that hard!  but its not wanting to work.

 

What am I missing?  Why will this not work?

 

Thank you

 

Doug

 

 

 

Link to comment

You can use the pcie number to stub instead of the whole card. This is what I have in my syslinux.cfg for 3 of my 4 ports.

 

xen-pciback.hide=(04:00.3)(04:00.2)(04:00.1)

 

If you have made the VM previous to adding the network cards, you might have to add them manually. At least I had to in my Ubuntu server VM. What is the output of the below command in the VM.

 

lspci

 

Link to comment
39 minutes ago, saarg said:

You can use the pcie number to stub instead of the whole card. This is what I have in my syslinux.cfg for 3 of my 4 ports.

 


xen-pciback.hide=(04:00.3)(04:00.2)(04:00.1)

 

If you have made the VM previous to adding the network cards, you might have to add them manually. At least I had to in my Ubuntu server VM. What is the output of the below command in the VM.

 


lspci

 

root@pbx:~# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:07.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:07.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:07.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:07.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
02:01.0 Ethernet controller: Red Hat, Inc Virtio network device
02:02.0 Communication controller: Red Hat, Inc Virtio console
02:03.0 SCSI storage controller: Red Hat, Inc Virtio block device
02:04.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
02:05.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
02:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
root@pbx:~#

Oh SNAP!  They have been pass through!  So off to figure out how to add them in to the OS so that it can see them.  Do you have any starting points for me?

 

Thank you!

 

Doug

Link to comment
31 minutes ago, doug.holcombe said:

root@pbx:~# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:07.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:07.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:07.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:07.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
02:01.0 Ethernet controller: Red Hat, Inc Virtio network device
02:02.0 Communication controller: Red Hat, Inc Virtio console
02:03.0 SCSI storage controller: Red Hat, Inc Virtio block device
02:04.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
02:05.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection
02:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
root@pbx:~#

Oh SNAP!  They have been pass through!  So off to figure out how to add them in to the OS so that it can see them.  Do you have any starting points for me?

 

Thank you!

 

Doug

 

You need to add them in /etc/network/interfaces. Looks like below.

 

auto eth0
iface eth0 inet dhcp

Most likely, your new interfaces will be eth1 and eth2. So just add two of the above and replace with the correct ethx.

 

For more info you can read more here

Link to comment
  • 1 year later...
On 2/25/2017 at 2:42 PM, saarg said:

You can use the pcie number to stub instead of the whole card. This is what I have in my syslinux.cfg for 3 of my 4 ports.

 


xen-pciback.hide=(04:00.3)(04:00.2)(04:00.1)

 

If you have made the VM previous to adding the network cards, you might have to add them manually. At least I had to in my Ubuntu server VM. What is the output of the below command in the VM.

 


lspci

 

So does this mean I can pass 2 of my 4 ports on a nic to my vm's and let unraid have the other 2 this way or do i still need a seperate nic to do this with and pass the whole care even though in my device list, the card has 2 ports in each of their own groups but with same vendor id.

Link to comment

If they are in separate groups, you can pass one through and let unraid use the others. 

If you use the method I mentioned, you use the PCI ID and not the vendor ID. This way you can pass hide which part of the card you want to pass through.

Only thing different to the normal stubbing, is that you have to manually add the device tag in the xml. You can't choose it in the other devices list. 

  • Like 1
Link to comment
1 minute ago, saarg said:

If they are in separate groups, you can pass one through and let unraid use the others. 

If you use the method I mentioned, you use the PCI ID and not the vendor ID. This way you can pass hide which part of the card you want to pass through.

Only thing different to the normal stubbing, is that you have to manually add the device tag in the xml. You can't choose it in the other devices list. 

Sounds like this is what I am aiming for...

 

 

This is the IOMMU group info for my nic, not sure if there is a way to figure out which corresponds to which port...

IOMMU group 13:	[14e4:1639] 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
                [14e4:1639] 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
IOMMU group 14:	[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
                [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

 

Would this be my append line then?

append xen-pciback.hide=(02:00.0)(02:00.1) initrd=/bzroot

 

Would this be what I need in my VM xml then???

<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
  </source>
</hostdev>

 

Link to comment
43 minutes ago, Repooc said:

Sounds like this is what I am aiming for...

 

 

This is the IOMMU group info for my nic, not sure if there is a way to figure out which corresponds to which port...


IOMMU group 13:	[14e4:1639] 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
                [14e4:1639] 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
IOMMU group 14:	[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
                [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

 

Would this be my append line then?


append xen-pciback.hide=(02:00.0)(02:00.1) initrd=/bzroot

 

Would this be what I need in my VM xml then???


<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
  </source>
</hostdev>

 

 

Your XML looks good and will pass through the iommu group 14.

 

Your append line it looks fine too.

 

But just as a side note many people (myself included until jonp told me a while ago) you can just add extra parameters to the end of the line. They don't have to be between the append and the initrd

 

so the line can look like this as you have it

append xen-pciback.hide=(02:00.0)(02:00.1) initrd=/bzroot

or just at the end

append initrd=/bzroot xen-pciback.hide=(02:00.0)(02:00.1)

they just have to be on the same line. I think it is less likely to make a typo have it at the end.

  • Like 1
Link to comment

I had found a post where someone posted a handy bash script to list your nic ports with link status in which i had plugged in to the router/switch and was able to see which was connected.

 

#!/bin/bash
for ETH in $(find /sys/devices/ -type d -iname "eth[0-9]"); do
  port=$(basename $ETH);
  export $(cat $ETH/device/uevent);
  link_status=$(cat $ETH/operstate);
  desc=$(lspci -d $PCI_ID| cut -d ':' -f3)
  pci_bus=$(echo $PCI_SLOT_NAME|sed -e 's#0000:##')
  echo -e "Found NIC: $port\n  Name:   $desc\n  PCI id:  $PCI_ID\n  PCI bus: $pci_bus\n  Status:  $link_status\n"
done

Can't recall the original author that posted this but credit goes to them.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.