***GUIDE*** Passthrough Entire PCI USB Controller


archedraft

Recommended Posts

This has been working great for me, today I rebooted my unraid server and added some memory to my VM. After making those changes the edits were gone from my go file and from the VM's XML. Any idea why that disappeared? I'm assuming that this doesn't need to be done with every reboot.

When you edited the memory the VM's XML would have removed any custom edits.  Limetech has said they will put a warning about that in a future version.  So that could be why the XML edits disappeared but does NOT explain the "go" file.
Link to comment
  • 3 weeks later...

Great guide, I was able to get this to work, but just want to point out a few things.

 

For me I wanted to pass in two top side chasis ports, which I found to be on bus 007 and on 00:16.2. The IOMMU group was:

 

/sys/kernel/iommu_groups/14/devices/0000:00:16.0
/sys/kernel/iommu_groups/14/devices/0000:00:16.2

 

When adding the controller in via the xml file, I needed to only add device 00:16.0 as adding both (00:16.2 as well) would cause an error. I assume it was enough to add 16.0.

 

Second, /usr/local/sbin/vfio-bind was not there by default. I just created the script: (hope this is what everyone else here is doing)

 

cat << EOF > /usr/local/sbin/vfio-bind
#!/bin/sh
modprobe vfio-pci
for dev in "\$@"; do
        vendor=\$(cat /sys/bus/pci/devices/\$dev/vendor)
        device=\$(cat /sys/bus/pci/devices/\$dev/device)
        if [ -e /sys/bus/pci/devices/\$dev/driver ]; then
                echo \$dev > /sys/bus/pci/devices/\$dev/driver/unbind
        fi
        echo \$vendor \$device > /sys/bus/pci/drivers/vfio-pci/new_id
done
EOF
chmod +x /usr/local/sbin/vfio-bind

 

And then adding this to the go file?

 

Lastly, I'm assuming this would work with a usb 3.0 controller, no caveats there?

 

Link to comment

 

 

When adding the controller in via the xml file, I needed to only add device 00:16.0 as adding both (00:16.2 as well) would cause an error. I assume it was enough to add 16.0.

 

Second, /usr/local/sbin/vfio-bind was not there by default. I just created the script: (hope this is what everyone else here is doing)

 

And then adding this to the go file?

Are you running unRAID 6.1? I haven't tried yet.

 

 

Lastly, I'm assuming this would work with a usb 3.0 controller, no caveats there?

Yeah I currently pass an USB 3 controller to my Windows 10 VM.

Link to comment
  • 3 weeks later...

Can someone explain what I did wrong in the first section?  The second code section works.

The difference is in the "  <qemu:commandline>" section.

 

What the guide told me to do:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>TV</name>
  <uuid>33b9695d-acfa-0641-8fa6-769acc37f2b3</uuid>
  <metadata>
    <vmtemplate name="Custom" icon="windows.png" os="windows"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.3'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/cache/Virtual Machines/TV/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Downloads/PC System/OS/Windows 10/en_windows_10_multiple_editions_x64_dvd_6846432.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Downloads/PC System/OS/virtio-win-0.1.102.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:6b:3c:3f'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/TV.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1b.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
<qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:14.0,bus=root.1,addr=00.1'/>
  </qemu:commandline>
</domain>

 

 

 

This appears to work:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>TV</name>
  <uuid>33b9695d-acfa-0641-8fa6-769acc37f2b3</uuid>
  <metadata>
    <vmtemplate name="Custom" icon="windows.png" os="windows"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.3'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/cache/Virtual Machines/TV/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Downloads/PC System/OS/Windows 10/en_windows_10_multiple_editions_x64_dvd_6846432.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Downloads/PC System/OS/virtio-win-0.1.102.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:6b:3c:3f'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/TV.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
     <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on'/>
<qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:14.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1b.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
  </qemu:commandline>
</domain>

Link to comment

Thanks for the excellent guide. [sOLVED]

 

Having a few problems passing through the controller however.

 

I mapped out the ports as suggested. The ports I wish to pass through are all on Bus 4.

 

The output of readlink

 

# readlink /sys/bus/usb/devices/usb4

 

is

 

../../../devices/pci0000:00/0000:00:12.0/usb4

 

I have updated the XML and vfio bind. Connected usb keyboard (did not get power) and confirmed that it did not show up in lsusb.

 

Using virsh to start the domain I get the following error.

 

virsh # start windows81
error: Failed to start domain windows81
error: internal error: early end of file from monitor: possible problem:
2014-12-22T16:45:59.153076Z qemu-system-x86_64: -device vfio-pci,host=00:12.0,bus=root.1,addr=00.1: vfio: error, group 6 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2014-12-22T16:45:59.153139Z qemu-system-x86_64: -device vfio-pci,host=00:12.0,bus=root.1,addr=00.1: vfio: failed to get group 6
2014-12-22T16:45:59.153172Z qemu-system-x86_64: -device vfio-pci,host=00:12.0,bus=root.1,addr=00.1: Device initialization failed.
2014-12-22T16:45:59.153206Z qemu-system-x86_64: -device vfio-pci,host=00:12.0,bus=root.1,addr=00.1: Device 'vfio-pci' could not be initialized

 

going back and entering

 

lspci | grep USB

 

I see

 

00:12.0 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.5 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:16.0 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: AMD/ATI [Advanced Micro Devices, Inc.] SB7x0/SB8x0/SB9x0 USB EHCI Controller
02:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

 

I am guessing that I also have to vfio-bind 00:12.2 as well as 00:12.0 but when I do this by editing the XML file and vfio-bind I get the following

 

virsh # start windows81
error: Failed to start domain windows81
error: internal error: early end of file from monitor: possible problem:
2014-12-22T16:57:26.634081Z qemu-system-x86_64: -device vfio-pci,host=00:12.2,bus=root.1,addr=00.1: PCI: slot 0 function 1 not available for vfio-pci, in use by vfio-pci
2014-12-22T16:57:26.634143Z qemu-system-x86_64: -device vfio-pci,host=00:12.2,bus=root.1,addr=00.1: Device initialization failed.
2014-12-22T16:57:26.634199Z qemu-system-x86_64: -device vfio-pci,host=00:12.2,bus=root.1,addr=00.1: Device 'vfio-pci' could not be initialized

 

currently my XML file qemu section looks like this

 

<qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:12.0,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:12.2,bus=root.1,addr=00.1'/>
  </qemu:commandline>

 

Not sure what I am doing wrong here. Help and thanks

 

[sOLVED]

 

Removed

 

<qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:12.2,bus=root.1,addr=00.1'/>

 

from XML but left the Go file looking like this

 

/usr/local/sbin/vfio-bind 0000:04:00.0 0000:04:00.1 0000:00:12.0 0000:00:12.2

 

The VM starts and the Microsoft all-in-one keyboard trackpad that would not work for more than a minute or so without stopping now works great and is not having any instability issues. I can plug other USB devices into any of the ports that belong to the USB controller and they work hot plug style.

 

Thanks

 

I have this problem with the same microsoft keyboard, so I have opted to try passing the entire USB controller through, which I'm fine with.  The VM boots up, but I still am losing the keyboard after a few seconds after it (the keyboard) powers up.

 

I'd like to try the suggested fix below from Jude, but in 6.1 things are a bit different and the go file entry isnt apparently needed, so I'm not sure exactly how to do this. 

 

Here's the relevant section of my XML at the moment:

 

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.0,bus=root.1,addr=00.2'/>
  </qemu:commandline>

 

Any thoughts?  00:1d.0 is the USB controller in question.

Link to comment
  • 3 weeks later...

I was trying to get this to work, however I have my all my USB ports split between 2 different Bus, but they are both on the same PCI controller. Am I still able to pass-through the bus that does not have my unRaid USB attached to it, even though they are on the same PCI controller?

Link to comment

 

I was trying to get this to work, however I have my all my USB ports split between 2 different Bus, but they are both on the same PCI controller. Am I still able to pass-through the bus that does not have my unRaid USB attached to it, even though they are on the same PCI controller?

 

What is the output of

lspci | grep USB

Link to comment

 

I was trying to get this to work, however I have my all my USB ports split between 2 different Bus, but they are both on the same PCI controller. Am I still able to pass-through the bus that does not have my unRaid USB attached to it, even though they are on the same PCI controller?

 

What is the output of

lspci | grep USB

 

lspci | grep USB output is:

 

00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

 

And lsusb is:

 

Bus 004 Device 002: ID 8087:8001 Intel Corp. 
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 8087:8009 Intel Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 0bc2:3322 Seagate RSS LLC 
Bus 002 Device 002: ID 0bc2:3322 Seagate RSS LLC 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0781:5567 SanDisk Corp. Cruzer Blade
Bus 001 Device 005: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

 

I tested all my ports with a USB drive, and Bus 1 is my 2.0x2 USB ports, and Bus 2 is my 3.0x6 USB ports, with 4 on the back panel and and 2 on the front panel. I never did get any USB ports to show up on Bus 3 or 4 which both have their own PCI address. While Bus 1 and 2 share the same PCI address.

Link to comment

Ok so what is your output of:

readlink /sys/bus/usb/devices/usb1

readlink /sys/bus/usb/devices/usb2

 

Also can you show me what your  IOMMU Groups shows?

Go to your unRAID gui -> Tools -> System Devices -> IOMMU Groups

 

Just want to make sure I fully understand what is going on before I recommend the next steps.

 

Link to comment

Ok so what is your output of:

readlink /sys/bus/usb/devices/usb1

readlink /sys/bus/usb/devices/usb2

 

Also can you show me what your  IOMMU Groups shows?

Go to your unRAID gui -> Tools -> System Devices -> IOMMU Groups

 

Just want to make sure I fully understand what is going on before I recommend the next steps.

 

Sure thing, here is the output for IOMMU:

 

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/5/devices/0000:00:1a.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.2
/sys/kernel/iommu_groups/7/devices/0000:00:1c.3
/sys/kernel/iommu_groups/7/devices/0000:02:00.0
/sys/kernel/iommu_groups/7/devices/0000:03:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3

 

I have a fairly new Asus Z97 motherboard with i5. So it has VT-d, and the motherboard bios has a ton of options for xHCI and EHCI. Default it is set to pass all the USB ports to the xHCI controller. Would there be a way possibly to set it up so the 2.0 ports are on EHCI and 3.0 on xHCI, then that should split them up on different buses as well as different PCI addresses, as the 2 unused EHCI controller each have their own PCI address as well. Appreciate the help in finding it the solution for this!

Link to comment

So with the help of this guide and others I have essentially successfully passed through an onboard AMD USB3 controller and a NEC 720200? controller that worked well with ESXi. More testing is required.

 

A problem I have though (this is in Windows XP by the way) is that when I shut down the VM and then start it again (not restart) both the AMD and NEC controllers get a code 10, cannot start problem.

 

This seemed repeatable in limited testing.

 

The safely remove hardware option actually lists all passed through devices and the virt devices as well. It seemed (tested once) if I safely removed the USB controller before shutting down it worked when started back up. I seem to recall people having this problem with video cards at one point.

 

I suspect this is because FLR or something is not handled well. I know this implementation frees up devices when the machine is shut down so other machines could use them. I wonder if setting the managed option to "no" in the XML combined with blacklisting the device from unraid with that pci stub option would alleviate this problem.

 

Weirdly, my HVR-1850 capture card seems to have no issues.

Link to comment

A problem I have though (this is in Windows XP by the way) is that when I shut down the VM and then start it again (not restart) both the AMD and NEC controllers get a code 10, cannot start problem.

 

This seemed repeatable in limited testing.

 

The safely remove hardware option actually lists all passed through devices and the virt devices as well. It seemed (tested once) if I safely removed the USB controller before shutting down it worked when started back up. I seem to recall people having this problem with video cards at one point.

 

I would be interested in knowing what happens if you installed Windows 7 or 10 and tried passing through the controllers. I have never tried this with Windows XP so my thoughts immediately jump to that as a strong possibility. I have never had any issues with my controllers on Windows 7, 8, or 10 but that could very well just be my hardware...

Link to comment

I can check it out at some point, but honestly running these devices on Windows 7 isn't really part of the end goal. I only have a few licenses for that and the installs take up to much space for what I want them to do. But it would be interesting to know regardless.

 

I don't think I had any issues with the NEC card like this with ESXi but it wasn't possible to even pass through the onboard USB with ESXi.

Link to comment

I can check it out at some point, but honestly running these devices on Windows 7 isn't really part of the end goal. I only have a few licenses for that and the installs take up to much space for what I want them to do. But it would be interesting to know regardless.

 

I don't think I had any issues with the NEC card like this with ESXi but it wasn't possible to even pass through the onboard USB with ESXi.

I have one of two USB MB controllers (the other is for ESXi and unRAID flash drives) passed through with ESXi 5.0 on a Tyan S5512 MB and E3-1230 v1 CPU works great with my HVR-950Q USB tuners on my Windows 7 x64 VMs.  Haven't tried it with unRAID native yet but I will likely be trying it this winter some time.
Link to comment

So I did a few more tests last night and what I found was that my original assumption was not correct. When I started the server yesterday I was surprised to find that one of the controllers had a code 10 again. I had previously thought this was cleared up when rebooting the hypervisor.

 

I then ran a bunch of reboot, shutdown, restart tests with both controllers using a couple different USB flash drives on both passed through controllers. I had one weird situation where windows may not have yet polled the flash drive (had a drive present but said "please insert a drive" or something when I tried to open it immediately after boot) but no other problems at all. My running theory now is that if I have a flash drive installed on a passed through controller, unraid tries to do something with it on startup which causes the strange state. In theory the controller should just reset. In practice, maybe that doesn't always work.

 

I didn't run a lot of hypervisor reboot tests so I'm still not sure I've nailed it down. I would think adding the devices to the syslinux.cfg file under pci-stub.ids would prevent unraid from ever looking at them directly but I didn't have a chance to try that either.

Link to comment

Its even more specific, only one of my flash drives (USB3 Lexar) causes this problem if its installed prior to the VM starting. An older USB2 one is fine. I even tried adding the controllers to pci-stub but that didn't fix it either. I was going to say maybe unraid loads them first or something and they aren't getting cleanly unbound on VM start, but given it still happens when its stubbed and doesn't happen with an old sandisk cruizer that doesn't really add up to me.

 

I ran a lot of tests and on both the NEC pci-e card and the onboard AMD USB controller it has the same effect on both and seems very reproducible once I figured out what was causing it. Since the AMD controller has two sets of ports only the one the flash drive is installed to gets a cannot start code 10.

 

Weird. Now I should try this on bare metal to rule that out and maybe try a couple other devices. I wouldn't be surprised if esxi has the problem as well, I didn't own this flash drive when I was testing on that.

Link to comment

My money is still on the fact that you are running Windows XP. Especially since it is happening with a USB3 drive. I would check your motherboard manufacturers website and see if they have any XP drivers for USB3. I know that in a VM your emulating a lot of that but it couldn't hurt to try. Could you setup a test Windows 7 VM (don't put in the Windows key) and just see if that fixes it? You should have 3 days to test the VM before Windows makes you register.

Link to comment

 

Well, I installed Windows 7 and preliminary tests show it behaves pretty much the same as XP. Code 10 if something is connected at startup.

 

Hmm, and it only happens when you use your one flash drive? Or do other devices give the code 10 if they are connected?

 

It also happened with a USB3 hard drive, not not a usb2 flash drive.

 

Well, I installed Windows 7 and preliminary tests show it behaves pretty much the same as XP. Code 10 if something is connected at startup.

Have you tried to hide the controller from unraid with pci-stub.ids yet?

 

Yes, I thought for sure that would fix it. I confirmed in the logs it was being assigned to pci-stub as well.

Link to comment

So I haven't played with this much recently. I still have that problem. I got ESXI working and tried to reproduce there and I could not. Both onboard and NEC card seem to work prefectly for me once I set it up correctly.

 

I noticed that when VMs are down/off that use the passed through controller with the samsung external harddrive with ESXI the activity light remains off and the drive is off (this drive is weird and only turns on if it detects it is connected or something. With unraid, the drive shuts off when a VM shuts down but then it will come back on after. This leaves me leaning towards the idea that the host screwing around with the controllers and leaving them in a gummed up state so that they don't always work. I found a post about some one doing passthrough with an AMD videocard and having the passthrough messed up after he loaded the AMD drivers on the host (he had two AMD cards) even though he had assigned the device to pci-stub. He uninstalled the radeon driver from the host and it solved it. Its not the same thing but its sort of close.

 

My motherboard is gigabyte so that weird on/off charge feature that has no options in the bios also seems like something I could blame.

 

All I can think now is to try blacklisting the xhci module. I have yet to see the problem with only USB2 devices attached although I've only tried a few. I think I just need to add:

append xhci_hcd.blacklist=yes initrd=bzroot to syslinux.cfg

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.