Up and Running - Dual Xeon with GPU passthrough

[colincoates]

At last my project is almost done...

 

My poor little Microserver was getting pushed to it's absolute capacity (N54L with 4GB Ram running 2 Ubuntu, 1 Freebsd & 1 Windows7 VMs as well as the owncloud & Guacamole dockers). Needless to say it was a bit sluggish.

 

Inspired by this post http://lime-technology.com/forum/index.php?topic=40349.0 I started to do a bit of googling.

There is a very good blog on hacking the z800 motherboard into a non-standard case and getting it to run on an ATX PSU http://andybrown.me.uk/wk/2014/11/01/z800/

 

This is my kit list.

 

Nanoxia Deep Silence 6 Case

HP Z800 Workstation Motherboard (Revision 2 - so no X56xx Xeons for me  )

Matched Pair Intel Xeon X5570 2.93GHz

Transcend 8GB (2x4GB) DDR3 PC1333 PC3-10600 1.5V ECC DIMM

ATX 24P + IDE 4P Molex to 18P + 10P Converter

1GB XFX HD 5450 Silent GPU

2X Zalman CNPS10X-Optima Shark's Fin Blade CPU Cooler

850W NZXT Hale82, Hybrid Modular, 80 PLUS Bronze

EVGA SuperNOVA 1.3KW Gold PSU (The 850W wouldn't start it so it got RMAed, also the first EVGA was a DOA)

5 Port Belkin F5U220qjLI  USB 2.0 PCI Card

256GB SanDisk X110 Enterprise Class 24x7, 2.5" SSD

 

I also transferred the following from my MicroServer.

3x HGST 4TB 7.2K SATA

2x Samsung 1.5TB 5.4K SATA

 

The physical build was quite challenging - especially drilling the motherboard tray, balancing it on a couple of M4 nuts and washers (used as spacers) and getting the PCI slots lined up with the back of the case.

 

Moving this case is also a nightmare - it is massive. I read reviews before I got it and they all said it was huge, but I didn't really realise how big until it was in front of me - it's heavy too. Not fun carrying it down 2 flights of stairs fully built.

 

The first thing I did when if finally came to life was put a copy of Windows 8.1 on it leave it running prime95 overnight to make sure it wouldn't die. I then ran a passmark performance test - 1511 (the GPU let it down a bit - but no matter I only want this GPU for Kodi)

I then put the Unraid trial on it and built a virtual Windows 8.1 VM with all the virtio drivers (with only 6 GB of ram and the first 2 cores from each CPU) - passmark this time of - 1320

I'm happy enough with that.

 

I stopped docker and all the VMs on my Microserver, copied everything from my HDD cache to the array, moved the disks to my z800 frankenserver, put my licenced usb key in and fired it up. It worked!!! Just needed to change the cache drive to the new SSD and start the array. Copied the VM's and docker image to cache and started the VM's.

The only problem I had here was a message from docker telling me to recreate the Image because of something in one of the Betas - I did this and redownloaded the dockers, pointed them to their old config directory and they all worked.

 

I built a kodiubuntu VM and passed it the GPU (so easy with the GUI now!), passed it a couple of 9p mount points and that worked too!!

I couldn't believe how easy and awesome unraid is!!

 

This may seem like overkill for a nas/mediacentre, but I have big plans for the future...

 

I've got some more memory on the way (48GB worth!) and I'm keeping my eye on ebay for any decent graphics cards for reasonable prices.

I want to build a Windows machine for a bit of light gaming and get this - I want to virtualise ESXi inside KVM, I've read that it can be done and it will mean that I can get rid of my old IBM x3650 that I use to lab things for work.

 

Wish me luck, here are some pics and screenshots.

 

My initial testing, after spending a couple of hours tracing a fault with the custom power cable.

 

The physical windows 8.1 passmark.

 

The virtual windows 8.1 passmark (only 6GB of ram and half the available cores)

 

After tidying the cables and mounting all the drives.

 

Up and running.

 

VMs and Docker apps happily chugging away.

 

I've really enjoyed this build and looking forward to completing it soon.

If anyone has any suggestions on running ESXi as a KVM VM I would be very interested in any pointers.

 

 

[jrdnlc]

Impressive! I'm thinking about running Windows 10 VM but not sure what I would use windows for lol

 

Will you be running a Plex Server? I wonder how many streams all that power can handle

[colincoates]

Impressive! I'm thinking about running Windows 10 VM but not sure what I would use windows for lol

 

Will you be running a Plex Server? I wonder how many streams all that power can handle

 

Thanks!

I use Kodi at the moment for viewing media (local via HDMI - don't have much of a need for streaming at the moment), I tried plex on my old media centre (old dell laptop) about a year or so ago and didn't much like it - I might give it a try again now that I've got a bit of horsepower and my media collection is a bit better organised (Thanks to Sickrage and Couchpotato) 

The dual Xeons have a cpu mark of 9679, so I would think 4x 1080p streams would be possible - just a guess though, I've never tried it. 

 

 

 

[billington.mark]

Hi Colin,

 

Just realised you were inspired by my post (you're welcome!) haha.

 

Ive just put together a very similar build (z800 mobo etc), but im having issues passing through a gpu to any VMs.

did you change anything in the BIOS or to anything else to get the passthrough working?

 

Mark

 

EDIT: Got this working. Had an old old x600 graphics card installed temporarily getting things up and running which stopped any passthrough working.

[jonp]

Colin, you mind if I post about this build and your success on our blog?  This is awesome!!!

[WeeboTech]

Nice!  I'll be watching this thread for conversation on the ESX via KVM!  Now that's an interesting idea.

[garycase]

It will definitely be interesting to see the results of virtualizing ESXi in KVM !!

 

I'd have thought the other-way around would be a more likely scenario ... i.e. running UnRAID as a VM within ESXi and then running some VM's within KVM on UnRAID.    This would let UnRAID be shut down without disturbing other ESXi VM's.

 

 

[pepper]

I'm currently trying to do this on a z800 with a GT 720 and Radeon 6670, but I am getting a Vfio-pci error.

[colincoates]

Colin, you mind if I post about this build and your success on our blog?  This is awesome!!!

 

Hi Jonp,

Sorry for not getting back to you sooner, I hadn't turned notifications on.

I'd be happy for this to be featured in your blog. I've actually made some changes over the last couple of months (details in my signature).

I've getting into gaming more than I thought I would and not had any performance problems.

Thanks,

CC

[colincoates]

Nice!  I'll be watching this thread for conversation on the ESX via KVM!  Now that's an interesting idea.

 

Hi WeeboTech,

I've actually not been using my virtual ESXi machines as much as I thought I would, but I did get them working.

There was a bit of messing around to hide some cpu features.

  <cpu mode='host-model'>
    <model fallback='forbid'>Nehalem</model>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>

And also some extra qemu commands.

  <qemu:commandline>
    <qemu:arg value='-machine'/>
    <qemu:arg value='vmport=off'/>
    <qemu:arg value='-enable-kvm'/>
  </qemu:commandline>

There was also no vfio drivers for ESXi, so I had to fall back on e1000 nics.

Overall they work pretty well, but as I say, I've not been using them as much as I thought I would (I just use KVM to do virtualization).

Thanks,

CC