Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

  1. #1
    Join Date
    May 2008
    Beans
    93

    Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    I have a stock standard Xenial KUbuntu install (no PPAs or debs) with KVM-Qemu-Libvirt (and LXC-LXD)

    Currently I have kvm VMs running windows with an AMD gpu. These run successfully, no problems

    I've now acquired an nVidia GPU and wish to pass it through. I need to use the new "kvm hidden" and "hyperv vendor_id" strings to ensure the nVidia drivers load successfully.

    On my newer VM created on Xenial I can use the "kvm hidden" string but not the "hyperv vendor_id" string

    on my VM created under trusty (and migratde to Xenial) I cannot use either string

    when I check the domaincommon.rng file I find support for "kvm hidden" but not "hyperv vendor_id" which seems reasonable since the vendor_id clause was added to libvirt in version 1.3.3 and Xenial seems to have 1.3.1

    How do others address this issue ... I see references to people using these clauses on Xenial. Are you installing newer versions of qemu and libvirt via PPAs .... or have I missed something ?

    thanks for your help

  2. #2
    Join Date
    May 2008
    Beans
    93

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    Progress: I've managed to "upgrade" the Trusty VM to Xenial format.
    Need to use "virsh edit" to change
    Code:
      <os>
        <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
        <loader type='rom'>OVMF_[vm_name]</loader>
        <boot dev='hd'/>
        <bootmenu enable='yes'/>
      </os>
    To
    Code:
      <os>
        <type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
        <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
        <nvram>/var/lib/libvirt/qemu/nvram/[vm_name]_VARS.fd</nvram>
        <boot dev='hd'/>
        <bootmenu enable='yes'/>
      </os>
    And create the VARS.fd file using
    Code:
    sudo cp /usr/share/OVMF/OVMF_VARS.fd  /var/lib/libvirt/qemu/nvram/[vm_name]_VARS.fd
    where [vm_name] is the name libvirt knows your vm by

    then when the vm is first started you'll get a UEFI error which will eventually (after 3-5 failed restarts) lead to Windows Boot Repair where you can select Advanced and UEFI options. The next restart should work ok after that (seems to rebuild the UEFI vars)

    Then we can add the necessary <hidden state='on'> clause using "virsh edit [vm_name]"
    Code:
      <features>
        <acpi/>
        <hyperv>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='4096'/>
        </hyperv>
        <kvm>
          <hidden state='on'/>
        </kvm>
      </features>
    Which still leaves me without the necessary <vendor_id state='on' value='whatever'/> clause, which libvirt (virsh edit) rejects
    Last edited by redger; September 2nd, 2016 at 11:32 AM.

  3. #3
    Join Date
    Sep 2012
    Beans
    579
    Distro
    Ubuntu Gnome 17.04 Zesty Zapus

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    Hey redger, its been a while since I've seen you here, hope all is well?

    I remember reading about the new hyper-v fixes for nVidia cards and went on a hunt as to where I read about it, turns out it was in AW's slide deck for the recent KVM forum (http://awilliam.github.io/presentati...rum-2016/#/4/2)

    I've not seen any chatter myself about being able to implement this on 16.04 stock binaries, like you my libvirt is only 1.3.1 although the qemu version should be okay. I was thinking it might be possible to use custom <qemu:commandline> tags in the xml like in the go old days but I don't even think that would work as libvirt won't allow you to remove the CPU config and the new hyper-v tags need to be put into the -cpu line? Like you I'm still using the kvm hidden state hack for my GTX 970.

    FWIW, the ovmf changes you've needed to make look like they're because you've migrated from a consolidated ovmf+vars image to a split image, which I think is as a result of the libvirt upgrade from Trusty to Xenial and its handling of bios/uefi loaders. You possibly could have avoided that by sticking with the OVMF-pure.fd and just removing the nvram line, but TBH I'm specu-guessing here

  4. #4
    Join Date
    May 2008
    Beans
    93

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    hi mate, yes good to hear from you again

    You're correct about changing the OVMF setup ... I needed to do that in order to enable the KVM Hidden clause, libvirt was rejecting it otherwise (I didn't track through all the templates but it's clearly a difference between the "trusty" template and the "xenial" template as defined through the machine clause)

    The best reference source these days is probably the Arch wiki https://wiki.archlinux.org/index.php...to_Windows_VMs. There was extensive discussion of the wiki entry via the mailing list as it was being constructed.

    I had toyed with the idea of direct qemu clause, but i'd rather not if I can help it - it's generally better to let libvirt manage everything to ensure consistency and stability. I may yet experiment.

    It's probably possible to run windows with nvidia using something similar to my current setup, but I would probably need to remove the hyperv clauses ... which would have performance consequences I would rather avoid.

    It's also possible to use a PPA to obtain the latest virtualisation modules eg. https://launchpad.net/~jacob/+ppa-packages ... but I'm trying to retain a "pure" host at this stage, it's running about 6 LXD & LXC containers plus 4 KVM vms so stability is important hence the desire for a "vanilla" solution.

  5. #5
    Join Date
    Sep 2012
    Beans
    579
    Distro
    Ubuntu Gnome 17.04 Zesty Zapus

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    I agree with your sentiments about keeping away from custom qemu lines in the libvirt xml, I ditched them as soon as I was able! Appreciate the link to the arch discussion, will take a longer read.

    Regards the performance impact of removing hyperv, I can honestly say I've not noticed but then having never been able to run Windows with a passed through nVidia with hyperv enabled I've got zero comparability. I can however say I game away on my 970 without hyperv, albeit a casual Windows gamer...I prefer the laziness of my Xbox

    Regards the PPA you linked for updated virt packages...I caution you to consider this you last resort, I stopped using it a long time ago now as some of his builds were broken and there was no easy resolution. ppa-purge does work but not with libvirt, there is something broken in the packaging which results in a completely borked reversion and its a manual job from here, re-encountered this as a test case in recent weeks. The ppa-purge problem isn't Jacob's doing and he's a one-man-band I believe so not really in a position to support the masses with his builds.

  6. #6
    Join Date
    May 2008
    Beans
    93

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    Progress: I managed to pass the vendor_id parameter through BUT only by adding a qemu argument, not thru libvirt itself. A bit of a hack, but it seems to work.

    Added this to the end of the libvirt definition

    Code:
      <qemu:commandline>
        <qemu:arg value='-cpu'/>
        <qemu:arg value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv_vendor_id=sometext'/>
      </qemu:commandline>
    Windows now starts under libvirt control (well sort of, since the cpu parameters are not handled via libvirt, they're over-ridden by the above code)

    I couldn't just add "-cpu hv_vendor_id=" directly in the above format, I had to override ALL the cpu parameters rather than just expecting to be able to add an additional one. Not a great solution, but perhaps workable for the moment.

    Does anyone have a better solution ?

  7. #7
    Join Date
    Sep 2012
    Beans
    579
    Distro
    Ubuntu Gnome 17.04 Zesty Zapus

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    Tested this and I *believe* it has worked although the qemu command libvirt executes includes two -cpu parameters? Based on the fact that my guest started fine and the nVidia drivers loaded I am assuming the later -cpu parameter overrides the former? Do you know of anyway to confirm if the hyper-v enlightenment's are actually present from within the guest?

    FWIW, I ran a Heaven GFX benchmark pre and post change, the end results were the same so I suspect the performance improvements hyper-v brings must manifest under other circumstances.

  8. #8
    Join Date
    Jan 2015
    Beans
    7

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    I was just retrying to get this working after asking on vfio-users to no avail and what a coincidence. I tried adding the qemu:commandline params under the libvirt but my 750Ti still gives me code 43 on windows 10 (passthrough works fine on arch linux guest). This is even though it seems like its passing the command line flags correctly. I think I'll just wait for the release of Xenial where libvirt is finally upgraded from 1.3.1 to resolve this issue.

  9. #9
    Join Date
    May 2008
    Beans
    93

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    @sye737 .... have you also tried without the hyper-v enlightenments ie. do NOT specify the "hv_" parameters anywhere

    Remove the Hyperv section from the "Features".
    Code:
      <features>
        <acpi/>
        <kvm>
          <hidden state='on'/>
        </kvm>
      </features>
    also remove hypervclock from the clock section
    Code:
      <clock offset='localtime'>
        <timer name='rtc' tickpolicy='catchup'/>
        <timer name='pit' tickpolicy='delay'/>
        <timer name='hpet' present='no'/>
        <timer name='hypervclock' present='yes'/>
      </clock>
    And modify the additions at the end, to also remove "hv_" elements
    Code:
      <qemu:commandline>
        <qemu:arg value='-cpu'/>
        <qemu:arg value='host,kvm=off,hv_vendor_id=sometext'/>
      </qemu:commandline>
    Does the driver work now ? or still code 43 ?

    @KillerKelvUK ... you may be amused to discover that I don't actually have the new GPU yet (still "on a boat") so cannot test. OTOH I found this for verifying the use of hyper-v enlightenments https://www.spinics.net/lists/kvm/msg100181.html ... and some general background http://computerperformancebydesign.c...-introduction/ and https://www.linux-kvm.org/images/0/0...kvm_hyperv.pdf

  10. #10
    Join Date
    Jan 2015
    Beans
    7

    Re: Xenial KVM-QEMU-Libvirt VGA Passthrough nVidia Graphics To Windows

    Sadly I've tried almost every combination of hyperv tags (I've never had the hypervclock timer tag). I also tried the way recommended in vfio-users using a wrapper script to interdict the arguments so only one `-cpu` argument gets passed with the vendor_id set.

    Code:
    #!/bin/sh
    exec /usr/bin/qemu-system-x86_64 \
      `echo "\$@" | sed 's|hv_relaxed|hv_relaxed,hv_vendor_id=KeenlyKVM|g'`
    Every example of successful bypass of code 43 seems to come from an Arch/Fedora user that I'm assuming has newer libvirt/qemu.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •