Hi,
I haven't tried tinkering with the server yet but I think I got my answer. Thanks to meep for pointing out making the whole usb/pci controller passthrough the VM. Also this: http://lime-technology.com/forum/index.php?topic=36768.0. I'll post back if it works for me.
Edit:
I managed to make USB passthrough work with the above link. Except I used <qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.0'/> address is .0 instead of .1 in the first post. Working fine now except my LAN connection suddenly dropped after a few hours. I had to physically remove and reconnect the LAN cable from the back of the server to be able to reconnect. Anyway hopefully someone new who searches for the keyword "usb packet state check failed" will get this workaround.