The VGA passthrough club - did you succeed, and how?

**jamel-maison** · March 12th, 2015

I forget to mention a tuto that I will follow for next test :
http://www.pugetsystems.com/labs/art...14-04-KVM-585/

**jamel-maison** · March 17th, 2015

Hello,

after many unsucessuful try, I'll stay with present config :
DC7900 8GoRAM-250GoHD, integrated video intel, radeon 5450 passingthrough as secondary card. Performance equal native install (windows indice : 4,1/7,9)

to enable vt-d
http://jaredlxl.blogspot.com/2009/04...00-dc7900.html
to configure machine (majority copy-past)
http://www.pugetsystems.com/labs/art...14-04-KVM-585/
and of course :
https://bbs.archlinux.org/viewtopic.php?id=162768

My machine doesn't allow booting on integrated graphic with first slot x16 filled. It is mutually exclusive.

When I try to boot on a second card (ati or nvidia in place of integrated card), I encountered BSOD with radeon passingthrough and a code 43 with nvidia passingthrough. From my reading, error disapeared with qemu 2.2, which I tried. Unfortunally (my fault, I don't know how do properly), it didn't correct and, I was tired to experiment.

I leave card passingthrough in a x16 slot, but it is x4 electrical. Due to limitation of my machine. It is not a gamer pc.

For all, it cost to me 150 euros for HP + 35 euros for radeon. I've just added 2 DVB-S and in one machine, I have 3 (VDR - Windows Game - Linux server and Labs)

I obtained what I want (windows inside linux avoiding reboot with descent performances) and I will stay as is.

May be, I'll try again with more edge device.

Good luck for everyone who will try and I take my hat off to developer and all people involved.

**mogliii** · September 24th, 2015

Hello,

I want to give a short report that I succeeded to get VGA Passthrough working. It was not easy

1. I tried with an Asus A88X-Pro and an A10-7850K APU. First I tried with an ATI 6570, but could not succeed. IOMMU seemed to separate well (had to use the second pcie16 slot from top). I used OVMF bios. I tried everything but didnt work.

2. I got an MSI GTX750 Ti Gaming (as it has UEFI bios), and tried again with passthrough, but I could not resolve the Code 43 problem. Oddly, there was loss of video signal after boot screen. There is a thread in the vfio-user mailing list archive https://www.redhat.com/archives/vfio.../msg00302.html
I tried different versions of QEMU (up to 2.2), but nothing changed.

3. I got a new motherboard and CPU (ASRock Z87 Extreme6 and Intel Core i5 4460), and just switching the motherboard and adjusting the grub boot parameters, I could boot with passthrough and driver (340.52) worked.

4. I had some laggy mouse issues. As it turns out, this was due to the fact that I was running the VM from an LVM volume on a software Raid1. After moving to its own physical hard drive, the mouse lag is gone and the VM feels like native.

5. For USB, I got a AREA SD-PEU3V-2E2IL PCIe1x card with a Via VL800 chip. It seems all USB ports on the Z87 Extreme6 are in the same PCI device, so only all ports can be passed to the VM. First I thought this could resolve my laggy mouse issue. I first tried to pass the USB card to the VM, but there was some serious boot trouble, high CPU load, host instability and the card was not usable in the guest. So now I am passing through the onboard USB controller and using the USB card for the host.

6. I'm now using a KVM switch to switch between host and VM. TightVNC is also installed.

On Unigine Valley Benchmark, I get ~850 points in Ultimat HD setting.

This was very frustrating experience but happy it works now. But the initial problem was definitely the motherboard/APU.

**zexelon2** · September 24th, 2015

Okay, so I have gotten 98% of the way to what I want to achieve with VGA pass through. I have been using the latest Xen 4.5 on Ubuntu 15.04, my hardware is as follows:

CPU: Intel i5 (with Vt-d & Vt-x)
MB: MSI Z97 PC Mate (comes with Vt-d)
RAM: 16Gb DDR3
Video #1: ATI 7970
Video #2: ATI 7870 XT

Guest OS: Windows 7 x64

So far I have two booting installations of Windows 7 x64 with full graphics from the passed through videocards. Linux runs on the IGP provided by the intel chip. The ATI 7970 runs full speed and phenomenally well.

I am having major issues with the 7870 XT though. Something is screwed up between the drivers and xen for the 7870 XT. After a few minutes of running it corrupts the video memory, initially I thought it might be hardware related but after booting it in a native windows environment and stress testing it with several games, it ran perfectly fine. I then found that the 7870 XT also has the exact same issues with Qemu/KVM the solution there was to dissable hugepage support. I can not figure out how to do this with Ubuntu 15.04 so I just finished re-compiling the kernel to strip out all huge page support, havent been able to test it yet though to see if it worked.

As this whole experiment is to get a flight simulator working I only needed to pass through the mouse and keyboard to the main VM (the 7970). After that I used an awesome piece of software called Multiplicity that basically works as a software KVM and auto combines the two VMs as if they were simply an extended desktop (obviously you can not pass windows between them but the mouse and keyboard travel as if they were one and you can drag and drop files seamlessly). It is a paid app but its 19$ for 2 machines... Conversely there is another FOSS one called Synergy that claims to do the same thing but across Linux/Win/OSX. I have not tried this one yet, spending to much time trying to fix the corruption issues with the 7870XT.

So far I have been extremely happy with the performance of Xen, the flexibility of LVM2 and VGA passthrough of the 7970. If my re-compiled kernel does not fix the 7870XT I may just replace that card.

Once I get VM2 stabilized I will be testing the PV drivers for Windows 7.

On thing I absolutely love right now is thinprovisioning with LVM2, I "goldmaster'ed" my initial install of Windows 7 and now I am able to spawn/test/destroy VMs on a whim! If something works I just update the gold image then.

Very very happy with the current state of VGA passthrough and Xen. To be honest it has brought me back to Linux after many years of being a disillusioned Linux'ite.

Edit:

For ease of success stick to Intel hardware for CPU/MB/Ram and discrete ATI hardware for Graphics. I tried doing this several times in the past on laptop hardware, it is almost impossible due to technologies like optimus (from nvidia) really messing with passthrough. I have heard reports of success with Nvidia cards but only the Quadro series. Its odd, Nvidia has the best binary drivers for Linux by far... and the worst VM support... ATI has the worst Linux binary drivers but by far the best VM support.

**themainliner2** · May 24th, 2016

Originally Posted by zexelon2

For ease of success stick to Intel hardware for CPU/MB/Ram and discrete ATI hardware for Graphics. I tried doing this several times in the past on laptop hardware, it is almost impossible due to technologies like optimus (from nvidia) really messing with passthrough. I have heard reports of success with Nvidia cards but only the Quadro series. Its odd, Nvidia has the best binary drivers for Linux by far... and the worst VM support... ATI has the worst Linux binary drivers but by far the best VM support.

With the greatest of respect have you tried any AMD/Nvidia hardware? The Quadro myth needs putting to bed and I suspect all your Nvidia issues may be wholly laptop related. As for Intel having "by far the best VM support" I can't comment because I've only used AMD/Nvidia.

CPU: AMD FX 8350 Black Edition | AMD Phenom II X4 965 Black Edition
MB: ASUStek Sabertooth 990FX (r1.0)
RAM: 16Gb DDR3
Video #1: Nvidia Geforce 760 | Nvidia Geforce 750Ti
Video #2: Nvidia Geforce 750Ti | Nvidia Geforce 460 768MB

Guest OS: Windows 7 x64 | Windows 10 Enterprise
Host OS: Xubuntu 15:10 & 16:04 | antiX MX-15

I use Synergy and can report that it works seamlessly and effortlessly.

There are plenty of great resources on the 'net that can assist you in virtualising with VGA Passthrough using AMD/Nvidia hardware. I got very close to bare metal level performance and I will write up my method not because it's my own or very clever (I stand on the shoulders of giants), but in the hope that the way I explain the process might prove helpful when used next to others guides.

**heiko_s** · September 8th, 2016

3 1/2 years have gone since I started this thread. I started with Linux Mint 13 (Ubuntu 12.04, I believe), today I'm running Linux Mint 17.3 (Ubuntu 14.04). Back then I ran Xen with the xm toolstack, later I switched to Xen with xl. Since December 2015 I'm running kvm and pass through a Nvidia GTX 970.

Here my observations:

1. Both Xen and kvm are capable contestants, the preference depends on your hardware.
2. Xen has problems with some Nvidia cards, but nowadays one can patch the Nvidia driver inside Windows and the "incompatibility" is a thing of the past.
3. kvm seems to work fine with Nvidia, I had some issues with an AMD card in the past, but probably nothing serious.
4. Given the right hardware, Xen is actually much easier to set-up and configure for performance. Out of the box, Xen is better in handling different workloads. There aren't many "performance tweaking" options to begin with, and they are simply not necessary in Xen. No weird or undocumented options in the config file. The fact that Xen xm/xl uses a plain ole clear text config file is a bonus.
5. kvm seems to have gotten the upper hand in compatibility with graphics cards (though to be honest, I haven't tried any newer Xen releases). It's quite easy to set up, especially when using vfio-pci to bind the graphics card at boot (versus the pci-stub method) together with UEFI (and graphics cards with UEFI BIOS).
6. The BIG downside of kvm is its documentation, or the lack of it. Today kvm has zillions of features and options, many of which aren't really documented. Probably because of this, people use virt-manager, a GUI to configure virtual machines. EDIT: Redhat has some detailed documentation here. Unfortunately it's RedHat centric and only covers the virt-manager and libvirt utilities, not qemu. My how-to over at the Linux Mint forum uses a plain bash script to start the VM with the qemu command and options.
7. Talking about qemu: aside from its many options, developers changed some syntax. I haven't found a good documentation yet that gives a comprehensive description.
8. kvm performance: each little qemu command line option can have a huge impact on performance. The only thing that is not influenced by VM boot options is the passed through graphics card - that one seems to perform consistent. On my hardware changing the -machine pc to -machine q35 had a huge impact on drive performance, but also on CPU and general performance. It took me endless trials with different qemu options and system configurations as well over 100 benchmarks to tweak kvm in order to get to the performance level of Xen. In all fairness, others have reported that kvm gave them better performance (compared to Xen) to begin with.
9. I'm still not sure the HV extensions in kvm have any impact on Windows performance, and I need to check if Windows actually is "enlightened" (i.e. uses the HV extensions).
10. I'm currently using kvm. The reason is twofold: kvm allowed me to easily pass through my new Nvidia GTX 970 card (using the -cpu host,kvm=off option), and kvm plays nicely with the graphics card in the host, which means I can have an Nvidia GPU on the host with proprietary drivers, something that has been problematic under Xen (though I've read about a workaround). The installation was the easiest part and went quite smooth, performance tweaking is another story.

Conclusion

kvm is developing rapidly and kvm VGA passthrough has become the hipp thing to do. I do appreciate the hard work of the developers, but I do have a wish:
Get your documentation updated! The easiest way to debug kvm is to improve the documentation of qemu-kvm. Because in all honesty, kvm is littered with undocumented or poorly documented features. Many of the qemu command line options that surface when searching the net have no documentation, or the documentation is so poor that it's literally useless. Even the performance tuning sections within the documentation are often outdated, or simply misleading. The simplest task, for example finding feature lists for different releases, means wading through forum posts, mailing lists, etc.

Potential VGA passthrough candidates: don't give up on Xen. It has a lot going for, and if you have the right hardware, it will probably be easier to get a system running with good performance. I like the way Xen handles CPU resources - it balances nicely between VM(s) and host. As for passing through Nvidia cards (the consumer models), one can patch QEMU to disable PCI snooping, or patch the Nvidia driver under Windows (the latter will void the license to use the driver).

**patpro** · October 16th, 2016

Hi,

I'm going to share my experience with passthrough in general (VGA, USB, Sound). My setup is different from most of yours, because I'm using VMware ESXi 5.5 as my hypervisor. I've made this choice because I'm very comfortable using it (part of my job), and because I need to host a Mac OS X guest.
My hardware is:
- SuperMicro X10SRA-F
- Intel Core i7-5930K
- 32 GB RAM
- several SSDs and HDDs
- MSI Radeon R9 270X 2GB for Windows 7 pro guest
- MSI Radeon R9 270X 2GB for Ubuntu guest
- ATI Radeon HD 5770 2GB Mac Edition for Mac OS X guest

The motherboard sports 3 USB controllers, so I've dedicated one for each guest. Each USB controller has either 2 or 4 USB ports. Due to ESXi 5.5 limitation, USB 3 ports are seen as USB 2. No big deal for me.
It sports a Realtek HD Audio controller that I've setup to passthrough for Windows guest. Other guests can use a headset plugged in a USB soundcard for sound playback and mic.

This is working great for Mac OS X and Windows. I can use every software I want with full GPU power on Mac OS X and Windows, I can listen to my music form the Windows guest while typing this post on the Mac guest.

But unfortunately, I can't make the Ubuntu guest use the MSI Radeon R9 270X. I've tried 14 LTS, and 16 LTS, but none will succeed in using the ATI GPU. I've tried FreeBSD/PCBSD/TrueOS too, no success. Both OSes will detect the ATI GPU and properly identify it, but they fail at using it.

You can find more info about my setup here:
https://www.patpro.net/blog/index.ph...-of-the-setup/
https://www.patpro.net/blog/index.ph...system-part-3/