gridrunner

**VIDEO GUIDE** How to passthrough and boot from an NVMe controller for bare metal performance in unRAID

24 posts in this topic

Posted (edited)

 

Hi Guys, this Video is a tutorial on how to pass through an NVMe controller then boot a VM from it in unRAID.
This method uses a modified clover boot loader with the NvmExpressDxe-64.EFI added to it to be able to use the NVMe drive then boot.
You will also see how to copy your existing windows VM from a vdisk directly onto the NVMe drive.
This method of pass through will also allow unRAID to have access to the device when the VM isn't using it as it isn't stubbed from the host.

Hope you find it useful :D

 

How to passthrough and boot from an NVMe controller for bare metal performance in unRAID

 

Edited by gridrunner
1

Share this post


Link to post
Share on other sites

Posted (edited)

Hey Gridrunner,

 

First off thanks for the videos, they are super helpful and very relevant to things I'm interested in doing. 

 

I've tried your tutorial out, as it's something I've been trying to solve constantly for the last few weeks. I am getting stuck hover on the clover bootloader piece.

 

I tried using your clover img file to boot windows directly from my nvme disk, but it seems like there is some problem with that file for me, as it only loads to a grey screen that says clover boot manager, and no further. I started to dive down the road of building my own boot loader with the clover configurator, but quickly turned back as that program is daunting and somewhat out of my league at the moment. 

 

any suggestions on ways to get windows to load fully, or why i might be running into this issue in the first place?

Edited by psparks
great > grey, load > manager
0

Share this post


Link to post
Share on other sites

Brilliant video as usual :)

 

I am actually surprised that the non-pass through speeds are as high as they are. I am assuming that there is no way for this to work with OSX currently? It certainly wouldn't put me off grabbing an NVMe drive anyway with the the first set of speeds to be honest.

0

Share this post


Link to post
Share on other sites
1 hour ago, psparks said:

Hey Gridrunner,

 

First off thanks for the videos, they are super helpful and very relevant to things I'm interested in doing. 

 

I've tried your tutorial out, as it's something I've been trying to solve constantly for the last few weeks. I am getting stuck hover on the clover bootloader piece.

 

I tried using your clover img file to boot windows directly from my nvme disk, but it seems like there is some problem with that file for me, as it only loads to a grey screen that says clover boot manager, and no further. I started to dive down the road of building my own boot loader with the clover configurator, but quickly turned back as that program is daunting and somewhat out of my league at the moment. 

 

any suggestions on ways to get windows to load fully, or why i might be running into this issue in the first place?

 

@psparksUm, thats strange its not working for you.

You shouldn't even see the clover bootloader screen (the grey screen)

The config.plist in my file should boot the bootmgfw.efi from the windows EFI partition and boot windows.

Do you see any options on the clover screen to boot from? Any hard drives listed. If so you should be able to select it and press enter to boot.

Also, how have you installed the windows os to the nvme drive. Did you install it directly to the nvme or did you copy across the install from your windows vdisk like i did in the video?

It will not boot from a windows vdisk just stored on the nvme drive. It must be a proper partitioned disk.

Also one other thing to check. Your VM must be OVMF it cant be seabios. You have to be using the EFI boot windows 10.

Hope this may help.

0

Share this post


Link to post
Share on other sites

@gridrunner Yea it is super weird. I followed the tutorial to the T, converting a working vdisk using the easeus todo software, then formatting the leftover partition with unassigned devices.

 

On the grey screen there are no machines to boot from, just the clover icon.  the VM is also ovmf. 

 

I'm going to start fresh from the begining and try it again. I think my issue may have been the formatting of the disk going into the whole process. I'm going to try to do again it by formatting it with unassigned devices and not giving it any fs/partitions (it may have had a single partition going into the process). currently it's showing the file systems types in unassigned devices for those windows partitions that the software made, and I saw that it wasn't showing those in your video.

0

Share this post


Link to post
Share on other sites

Posted (edited)

You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install.

 

The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. 

We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting.

 

So basically, build the VM (dont start it yet) and pass through the NVMe device:

IOMMU group 41
	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)

would be:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

  • Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file
  • open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM.
  • reference the new files in XML (replace the default values):

 

<os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

 

Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done

 

Edited by billington.mark
2

Share this post


Link to post
Share on other sites

If you have the NVMe drive attached through a PCI card, would that make much of a difference? or do the PCI adapter cards basically pass through the NVMe card?

0

Share this post


Link to post
Share on other sites
2 minutes ago, planetwilson said:

or do the PCI adapter cards basically pass through the NVMe card?

 

PCI adapter is transparent to unRAID.

0

Share this post


Link to post
Share on other sites
11 minutes ago, johnnie.black said:

 

PCI adapter is transparent to unRAID.

 

+1. I use my NVMe device in a PCIe adapter

0

Share this post


Link to post
Share on other sites

Posted (edited)

1 hour ago, billington.mark said:

You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install.

 

The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. 

We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting.

 

So basically, build the VM (dont start it yet) and pass through the NVMe device:


IOMMU group 41
	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)

would be:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

  • Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file
  • open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM.
  • reference the new files in XML (replace the default values):

 


<os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

 

Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done

 

 
 
4

@billington.mark  Wow thats awesome. i didnt know that was in the ovmf bootloader now. i will try that asap. Guess i will be making an update to that video today :D

I guess we have to put

<boot order="1">  in the xml of the pci device

ie

<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
</source>
<boot order="1">
</hostdev>

 

 

Edited by gridrunner
0

Share this post


Link to post
Share on other sites
7 minutes ago, gridrunner said:

@billington.mark  Wow thats awesome. i didnt know that was in the ovmf bootloader now. i will try that asap. Guess i will be making an update to that video today :D

I guess we have to put

<boot order="1">  in the xml of the pci device

ie


<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
</source>
<boot order="1">
</hostdev>

 

 

 

Ive found that didnt make any difference after booting the VM (may have been fixed since i last tested though?). I've always had to manually edit the boot order in the actual VM by invoking the OVMF setup (del \ F2 i think?) after the first boot.

 

0

Share this post


Link to post
Share on other sites

@billington.mark

Thanks for the great tip on your passthrough technique. I got it working, but for some reason I need to edit the boot order every time the VM restarts. Is that expected or is there a way to ensure that boot order stays the same?

0

Share this post


Link to post
Share on other sites

Posted (edited)

Hi, I tried using the OVMF firmware. I find I don't get the windows symbol when windows loads it has the Tianocore logo now.

Doesn't make any difference though but anyone else have this?

 

Personally, I found it better to just replace just the OVMF_CODE-pure-efi.fd  in unraid.

Just copy it to /usr/share/qemu/ovmf-x64.  I found it works fine with only this file but you can copy all three if you want (OVMF-pure-efi.fd , OVMF_VARS-pure-efi.fd and OVMF_CODE-pure-efi.fd) The advantage is you can just create a VM without having to do any custom edits. And each VM gets its own NVRAM file as normal.

Just make a user script to run at the start of array and it will update the ovmf firmware each time the server boots.

I have attached my script below.  Along with the new OVMF firmware. Just put the files in a share and set its location in the script.

#!/bin/bash
#Set location of ovmf
ovmfstore=/mnt/user/test/ovmf
#copy to unraid
rsync -a $ovmfstore/* /usr/share/qemu/ovmf-x64

exit

 

 

ovmf.zip

user script.zip

Edited by gridrunner
0

Share this post


Link to post
Share on other sites

Posted (edited)

12 hours ago, gridrunner said:

Hi, I tried using the OVMF firmware. I find I don't get the windows symbol when windows loads it has the Tianocore logo now.

Doesn't make any difference though but anyone else have this?

 

Personally, I found it better to just replace just the OVMF_CODE-pure-efi.fd  in unraid.

Just copy it to /usr/share/qemu/ovmf-x64.  I found it works fine with only this file but you can copy all three if you want (OVMF-pure-efi.fd , OVMF_VARS-pure-efi.fd and OVMF_CODE-pure-efi.fd) The advantage is you can just create a VM without having to do any custom edits. And each VM gets its own NVRAM file as normal.

Just make a user script to run at the start of array and it will update the ovmf firmware each time the server boots.

I have attached my script below.  Along with the new OVMF firmware. Just put the files in a share and set its location in the script.


#!/bin/bash
#Set location of ovmf
ovmfstore=/mnt/user/test/ovmf
#copy to unraid
rsync -a $ovmfstore/* /usr/share/qemu/ovmf-x64

exit

 

 

ovmf.zip

user script.zip

 

This is much neater :) 

I dont have the no windows logo issue on the version im running... If you want the version i'm using, drop me a PM. 

 

Edited by billington.mark
0

Share this post


Link to post
Share on other sites

@gridrunner

I do have the logo 'issue' but everything else runs just fine, so seems like it's just an oddity.

 

and to answer my own question from earlier about not needing to change boot order every time you start the vm...I figured out you just need to remove the other boot options in the xml. So remove the vdisk, driver iso file, any dvdrom ect that were booting before your nvme drive and it'll go straight to the only thing left. 

1

Share this post


Link to post
Share on other sites
On 20/03/2017 at 9:29 PM, billington.mark said:

You can boot to windows with a passed through NVMe without using clover at all. I've been doing this for a while since the first 6.3 RC. Bear in mind this assumes you're doing a fresh install.

 

The only reason you cant do this natively is that Unraid ships with an older OVMF bootloader for VMs which is before it started supporting NVMe devices. 

We will just be downloading a referencing a different bootloader so we add support for direct NVMe booting.

 

So basically, build the VM (dont start it yet) and pass through the NVMe device:


IOMMU group 41
	[144d:a802] 81:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)

would be:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

  • Download OVMF firmware from here https://www.kraxel.org/repos/jenkins/edk2/. You want the edk2.git-ovmf-x64-x-xxxxxxxx.xxxxx.xxxxxxxx.noarch.rpm file
  • open with 7zip and drill down until you get to the edk2.git\ovmf-x64\ folder and pull out the OVMF-pure-efi.fd and OVMF_VARS-pure-efi.fd files and put them on an accessible share somewhere. Preferably on a cache only share (be sure to do both). in the xml below, Ive put them in /mnt/user/VMData/Windows10VM.
  • reference the new files in XML (replace the default values):

 


<os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/VMData/Windows10VM/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/VMData/Windows10VM/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

 

Boot the VM and go to the boot menu. Change the boot order to the DVD drive as top, NVMe as 2nd. Save and boot from the windows installation ISO. the NVMe drive should show as an available installation target in windows setup. Run through the installation as you normally would and you're done

 

@gridrunner

 

Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file.

 

New VM never been booted.

 

It seems that the VM template does not like the reference to "<boot dev='hd'/>"

 

This is on UNRAID 6.3.2

 

Any ideas?

 

Screen Shot 2017-03-25 at 16.45.55.png

0

Share this post


Link to post
Share on other sites
2 hours ago, Enver said:

@gridrunner

 

Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file.

 

New VM never been booted.

 

It seems that the VM template does not like the reference to "<boot dev='hd'/>"

 

This is on UNRAID 6.3.2

 

Any ideas?

 

Screen Shot 2017-03-25 at 16.45.55.png

2

 Yes you will have more than one boot in your xml thats why you are getting that error.

 You will have something like 

<os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/0a0c0f94-12fc-d6f4-f5ad-3f141f92f861_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>

But you will also have a boot order in the xml somewhere else.

<boot order='1'/>

You can only use one or the other so you will need to remove one from the xml

0

Share this post


Link to post
Share on other sites

Posted (edited)

7 hours ago, Enver said:

@gridrunner

 

Hello have attempted your method to pass-through the NVME drive but get the following error when I attempt to update the XML file.

 

New VM never been booted.

 

It seems that the VM template does not like the reference to "<boot dev='hd'/>"

 

This is on UNRAID 6.3.2

 

Any ideas?

 

 

 

just remove this part:

 

<boot dev='hd'/>

I tend to just set the boot device order in the actual VM in the OVMF bios (press F2 on boot i think?). There you can re-order boot devices as you would on a normal full fat PC BIOS.

As far as im aware, you cant set the boot order in the XML when one of your boot devices is a passed through PCIe device, like you can when you use VIRTIO disks. 

 

EDIT: post your VMs XML 

Edited by billington.mark
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

Copyright © 2005-2017 Lime Technology, Inc. unRAID® is a registered trademark of Lime Technology, Inc.