The VGA passthrough club - did you succeed, and how?

[heiko_s]

I can't do away with *******, but I hate dual-boot. I also need native (or near-native) graphics support in the Windows VM.

In the end I chose Xen to host Linux and a Windows 7 VM which uses a dedicated graphics card for native graphics performance. It works extremely well, with what I consider top-notch performance that doesn't stand (much) behind a native Windows installation on bare metal.

In fact, I'm so thrilled about it that I wonder why not every dual-booter is switching to virtualization. (I'm actually not that surprised since it was quite challenging to get it working.)

Here my questions:

Has anybody tried VGA passthrough?

If yes, which hypervisor/method did you use (Xen, KVM, VMware, etc.)?

How easy or difficult was it?

In the end I would like to know if there are perhaps better or easier ways to get it working. Thanks in advance for your feedback!

[xelra]

I once tried to setup a xen virtualization with Linux as Host and Win7 as guest with VGA passthrough, because I believed that I can get 99%+ native performace.
At the time tough it was very new and undocumented, so I gave up on it.

I know that to get it working you need hardware that supports both VT-x and VT-d.

My goal was to get fast switching on 2 desktops, the way you would have 2 full OSes booted simultaneously.
Can you confirm your Windows 7 installation behaves as native? What about other peripherals? Scanner, printer, gamepad, etc.?

Do you have a guide?

[heiko_s]

I once tried to setup a xen virtualization with Linux as Host and Win7 as guest with VGA passthrough, because I believed that I can get 99%+ native performace.
At the time tough it was very new and undocumented, so I gave up on it.

I know that to get it working you need hardware that supports both VT-x and VT-d.

My goal was to get fast switching on 2 desktops, the way you would have 2 full OSes booted simultaneously.
Can you confirm your Windows 7 installation behaves as native? What about other peripherals? Scanner, printer, gamepad, etc.?

Do you have a guide?

You indeed need suitable hardware. Here is what you need:

1. CPU with VT-d (Intel) or AMD-Vi (AMD) support. This feature is also called IOMMU. See http://en.wikipedia.org/wiki/IOMMU_hardware_list, or for Intel CPU's see http://ark.intel.com/search/advanced/?s=t&VTD=true.

2. A motherboard that supports VT-d. See the Wikipedia link above, as well as http://www.overclock.net/t/1338063/v...e-motherboards and http://wiki.xen.org/wiki/VTd_HowTo.

3. A motherboard BIOS that supports VT-d. I'm not joking here, one BIOS release may support VT-d, the next one doesn't. In some cases there is no way going back to the earlier BIOS that does support it. See links under 2.

4. A graphics card that supports VGA passthrough. In general, most modern AMD (ATI) graphics cards will work, but it's best to check here http://wiki.xen.org/wiki/XenVGAPasst...TestedAdapters and www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters.
Some Nvidia graphics adapters are also supported, namely the Quadro series of "Multi-OS" capable cards starting with the Quadro 2000 model upwards (the Quadro 600 for example isn't specified as "Multi-OS"). There are some more Nvidia graphics adapters that work, but first check the links above if it's listed.
The newer Xen 4.2 hypervisor supposedly improves graphics cards compatibility, though I haven't tried it.

It is MUCH easier if you have two graphics cards, or a CPU internal GPU (like on most new Intel desktop CPUs) and a discrete graphics card for passthrough.One will serve Linux, the other your Windows guest.

If your hardware doesn't meet the above, either replace the incompatible part, or forget the whole thing!

I've written a how-to for Linux Mint 13/14, which should likewise work with Ubuntu 12.04 and 12.10: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013.

Now to your questions.

My goal was to get fast switching on 2 desktops, the way you would have 2 full OSes booted simultaneously.

My PC works as if I have two PCs. I could hook up two screens, keyboards and mice and two people would be able to use it at the same time, with full video acceleration on both. Right now I switch from one to the other via KVM button and the input selector on my screen, but if I have to go forth and back a lot I run a remote desktop in Windows to connect to my Linux dom0, or vice versa. I use a SAMBA share under Linux to copy files between the two.

Can you confirm your Windows 7 installation behaves as native?

95% yes. Everything except the following two issues work as if I was running Windows directly on the hardware. The two issues I've found are:
I) Youtube video stops after a few seconds, hangs for a while, and continues, just to stop again after a few seconds. This is most likely a networking issue with the bridge, though I haven't figured out what exactly. Somehow Windows "looses" the DNS server, or access to it. One work around would be to pass through a dedicated NIC. I haven't tried it though.
II) I can't watch video using VLC when the video file is on a network drive. I believe this is the same network problem as above.
EDIT: Both problems are solved now using the latest GPLPV drivers under Windows.

Other than these two issues I haven't found any problem.

What about other peripherals? Scanner, printer, gamepad, etc.?

In addition to the graphics card, I'm passing 2 USB controllers through to Windows. Everything connected to these ports is fully controlled by Windows, with Windows native drivers.
Just an example how well this works: I use Windows for photo editing. To calibrate my screen I use a spectrophotometer (some device that measures light / colors) which is connected via USB. The software then shows different color patches on the screen which my calibrating device reads and reports back to the PC (via USB port). At the end, the software loads the corrected color curve directly into my screen via the DVI port of the graphics card. This can only work when the calibration software is able to communicate with the graphics card.
Other external devices I frequently use under Windows are:
- Keyboard and mouse are connected via KVM USB switch;
- CF card reader (to transfer photos from the camera's CF cards to the PC);
- External hard drive for backup - it's automatically detected by the backup program when I connect it to the USB port;
- USB audio controller - a low-cost ~5$ USB stick that works without any issues, it's permanently plucked in;
- USB flash drives - they work as expected.

I haven't set up my printer yet to work with Windows. But I can't see any reason why it shouldn't work. I have both a USB 2.0 and a USB 3.0 controller passed to Windows. The transfer speed is as expected. If you have an external HDD or memory card reader to connect, use a USB 3.0 controller - it makes a huge difference.

Good luck!

[hurenkam]

I can't do away with *******, but I hate Has anybody tried VGA passthrough?

Yes, I have a setup with 2 desktops, each running in their own virtual machine with passed through radeon4350 and usb cards.
Apart from that, a few other virtual machine servers run on the same hardware.

Which hypervisor/method did you use (Xen, KVM, VMware, etc.)?

I'm using Xen 4.2 at the moment, it seems to me that in general Xen is way ahead of the game when it comes to PCI Passthrough.

How easy or difficult was it?

Once i started to use the catalyst driver instead of the open source driver (which complains about not being able to find the bios), everything was pretty straight forward.
I installed ubuntu and windows7 in a vnc session and later installed Catalyst. Windows 8 was even easier, it installed a working vga driver out of the box.
Note that i just used pci-passthrough, not the advanced vga-passthrough patches.
Besides the vga adapter itself, i also passthrough the onboard audio of the adapter (hdmi), and an usb controller, so basically i have 2 seperate terminals, each running their own os (either Ubuntu/Windows7/Windows8).
All operating systems behave asif they run on native hardware, only difference is that the boot screen doesn't appear on my vga cards, because i chose to do only pci-passthrough. The boot window appears on the VNC session.

Note that as stated above, using the right CPU, Mainboard and VGA Card is essential here.
In my case: Intel Core i7 860, Asus P7P55D Evo, Asus EAH4350 Silent.

[heiko_s]

@hurenkam: Thanks for sharing your experiences with virtual machines and VGA/PCI passthrough. It really looks like Xen is ahead of the game, as I've seen very few reports about successful KVM or other passthrough.
Yes, having the right hardware is essential.

[heiko_s]

Update: Things seem to be changing lately. I see more and more people using kvm for VGA passthrough. While there are problems like host freezing when exiting/rebooting guest, which has a solution or work-around, some report that it's been easier to implement and has less problems with graphics adapters and drivers under the host OS (Linux).

So far it seems that Xen is still troubled with proprietary Nvidia driver support under dom0, and generally with VGA passthrough support of Nvidia cards. I don't know if kvm is better in that, but it may be worth a try.

Last not least, Xen has moved to 4.2.1 and xl toolstack and the xm toolstack has been discontinued. Some report problems with vga passthrough using the xl toolstack. I can't say as I'm still using xm / xend.

Note: My network / Youtube problems have been solved after installing the latest GPLPV drivers from http://www.meadowcourt.org/downloads/.

[jaripetteri]

Just drop in couple lines about my HTPC Home Virtual Server setup. I have build this same time as finishing my house during past year. Haven't have time to this but need to something to get my mind of house building time to time.

I have now i7 3770T CPU and Asrock Z68 Extreme 4 Gen 3 MB with 24GB memory. 256GB OCZ Vertex 4 SSD for root fs's and VM's plus 2x 3TB and 1,5TB HDD's for media and MythTV recordings. Also 2x 3TB WD My Book Elements USB 3.0 as backup drives. Most of media drives are bypassed to VM giving it Asmedia AS1061 SATA3 -PCIe card and USB 3.0 host from MB. Also two GPU (Radeon HD7750 & HD7770, did try GTS450 without success) are PCI Pass through to two VM's, another one as my main HTPC and another I'm planing to use as daily desktop (currently still using x220).

I'm using KVM as visualize platform and mainly Lubuntu and Ubuntu 12.04's. HTPC is Lunux Mint and Desktop Windows 7. With Windows I have some stability issues and therefore not using it daily yet. There is three VM's serving as MythTV backend, MySQL server and that NAS. Also DVB-C card is pass through to VM. During building I have had issues with IRQ's, getting short off those all a time and very few PCIe card is capable to pass through. Now I like to change my MB to something Z77 without PCI slots to get more PCIe slots since I'm running out of those and could use couple more...

And this all was possible because I run little web store and need VM server to test different things before to take those in use. I kinda got hooked to this VM thing... All of above was earlier on AMD x2 4580e server with single Linux Mint 12 installation.

[heiko_s]

Just drop in couple lines about my HTPC Home Virtual Server setup. I have build this same time as finishing my house during past year. Haven't have time to this but need to something to get my mind of house building time to time.

I have now i7 3770T CPU and Asrock Z68 Extreme 4 Gen 3 MB with 24GB memory. 256GB OCZ Vertex 4 SSD for root fs's and VM's plus 2x 3TB and 1,5TB HDD's for media and MythTV recordings. Also 2x 3TB WD My Book Elements USB 3.0 as backup drives. Most of media drives are bypassed to VM giving it Asmedia AS1061 SATA3 -PCIe card and USB 3.0 host from MB. Also two GPU (Radeon HD7750 & HD7770, did try GTS450 without success) are PCI Pass through to two VM's, another one as my main HTPC and another I'm planing to use as daily desktop (currently still using x220).

I'm using KVM as visualize platform and mainly Lubuntu and Ubuntu 12.04's. HTPC is Lunux Mint and Desktop Windows 7. With Windows I have some stability issues and therefore not using it daily yet. There is three VM's serving as MythTV backend, MySQL server and that NAS. Also DVB-C card is pass through to VM. During building I have had issues with IRQ's, getting short off those all a time and very few PCIe card is capable to pass through. Now I like to change my MB to something Z77 without PCI slots to get more PCIe slots since I'm running out of those and could use couple more...

And this all was possible because I run little web store and need VM server to test different things before to take those in use. I kinda got hooked to this VM thing... All of above was earlier on AMD x2 4580e server with single Linux Mint 12 installation.

Thanks for sharing your setup - this sounds really good! A little over a year ago I had a hard time finding documentation on VGA passthrough using kvm. Things have changed and I've found a few places to consult with, specifically http://www.overclock.net/t/1205216/g...#post_19770787.

I'm using Xen at the moment, with Linux Mint 14 as dom0 and Windows 7 Pro as domU, and am hesitant to switch to kvm, as everything so far has been rock solid. My installation uses the regular Linux Mint and Ubuntu repositories, no development packages or kernel compilations. From what I gathered kvm would need a newer kernel and perhaps some patches to get all the functionality?

What did impress me about kvm was what looks to be better support for graphics cards, in particular Nvidia cards. With Xen, one better sticks to AMD cards. I would have liked to run a Nvidia card under dom0, but Nvidia proprietary drivers don't seem to play well with a Xen hypervisor.

You've got a bunch of VMs running, 5 if I'm not mistaken. I'm curious about your CPU resource allocation? I'm really not familiar with kvm and have no idea how the kvm CPU scheduler works. The reason I'm asking is that in my setup I would need all CPU resources I can get for my Windows VM, but I do not want to pin these resources to the Windows VM. Is there a way to overbook CPU resources under kvm, for example assign 10 VCPUs to VM-1 and another 10 VCPUs to VM-2, while the host gets a guaranteed minimum VCPUs of 2 (I've got a 6-core CPU giving me 12 VCPUs)? Then if VM-1 doesn't need all the CPU resources they would be available under VM-2 or even the host.

My setup is somewhat flawed in that I use dom0 as my regular desktop, which I think isn't such a good idea. If I move to kvm I would want to change that, using a server distro for the host and running my Linux Mint desktop in a VM, same as the Windows VM.

Contrary to your approach of passing through most of the media drives (i.e. the controllers), I use LVM volumes for both the host and the Windows VM. So my host (dom0) has access to all the drives and backup is very simple (snapshot the volume, then dd and pigz/gzip to compress the data). Note: My Windows VM uses RAW storage on LVM. Here some Passmark benchmarks run within the Windows VM:

Xen with Windows VM using a Sandisk Extreme 120GB SSD:



Xen with Windows VM using two LVM striped WD20EARX 2TB HDDs:



Xen with Windows VM using LVM concatenated WD1001FALS and WD5000AAKS drives:

[GizmoChicken]

My setup is somewhat flawed in that I use dom0 as my regular desktop, which I think isn't such a good idea. If I move to kvm I would want to change that, using a server distro for the host and running my Linux Mint desktop in a VM, same as the Windows VM.

Following your incredibly detailed and well-written guide, I was able to set up GPU passthrough to a Windows 7 guest using a Radeon HD6670 card and Ubuntu 12.10 Desktop as dom0. Thanks for the guide!

But I too would prefer a server-based dom0, and so I've been experimenting with XCP 1.6, which is an open source version of XenServer. Using XCP 1.6, I've had good luck with GPU passthrough to a Windows 7 guest using the Catalyst 13.4 driver with a Radeon HD6670 card.

However, at least for me, getting GPU passthough to work with a Linux guest has been much more of a challenge, regardless of whether I use XCP 1.6 or Ubuntu 12.10 as dom0. Using XCP 1.6 as dom0, I've tried Ubuntu 12.04, 12.10 and 13.04 as guests, each with various display drivers. So far, of the nearly innumerable combinations that I tried, the only combination that has worked for me has been Ubuntu 12.04 combined with whatever default version of Catalyst (not the update) that is found in the Ubuntu repository.

So to reiterate, in addition to GPU passthrough to a Windows 7 guest, GPU passthrough to an Ubuntu 12.04 guest is also possible using XCP 1.6 as dom0. Passthrough to other Linux guests may also be possible using proper settings. I would greatly appreciate hearing about any successes others have had with GPU passthough to other Linux guests, whether using vanilla Xen, XCP or even KVM.

Regarding XCP 1.6, although passthrough of the primary GPU to a guest is possible, based on my experiences, I do not recommend doing so. Rather, I recommend devoting the primary video card to the XCP console. But the good news is that even a cheap card is sufficient for use as primary video card for the XCP console. And if the bios settings of your motherboard allow for selecting between PCI and PCIe as the primary boot graphics card (my Asus M5A99FX motherboard allows for this), to conserve your PCIe slots, I recommend devoting a cheap PCI card (not PCIe card) to the XCP console. In may case, an ATI Rage 8 MB PCI Video Graphics Card ($10), which only offers VGA output, works great for me as the primary graphics card devoted to the XCP console. (Other PCI cards, such as those having DVI and/or HDMI output my be more suitable for some applications.)

The XCP console provides an easy (menu-based) means for starting and stopping VMs. From its menus, you can also find lots of system information. However, XCP relies on "xe" rather than "xm" or "xl" for management. And unfortunately, "xe" commands can be a bit cumbersome. So many prefer a GUI based approach for managing more complicated tasks in XCP, with the Windows-based XenCenter currently being the most popular option. (XenCenter can be run from a network-attached laptop.) Although XenCenter isn't open source, it is, at least as of the time of this writing, freely downloadable from Citrix after creating a free Citrix account. (XenOrchestra is an open source web GUI that is currently under development, but not yet ready for production.)

Even if you use XenCenter for VM management, you'll still need to use "xe" commands on limited occasions, such as for setting up PCI passthrough to USB controllers, etc. The PCI passthrough command in "xe" takes the following form:

xe vm-param-set other-config:pci=0/<pci-id#0>,1/<pci-id#1>,2/<pci-id#2>,n/<pci-id#n>  uuid=<uuid>

The <uuid> for a given VM is obtainable from XenCenter and each <pci-id#> can be obtained using the familiar "lspci" command after connecting to XCP, such as via ssh. Here's an example using an "xe" command for passthrough of seven usb controllers to one VM:

xe  vm-param-set  other-config:pci=0/0000:00:12.0,1/0000:00:12.2,2/0000:00:13.0,3/0000:00:13.2,4/0000:00:14.5,5/0000:00:16.0,6/0000:00:16.2  uuid=0370e4b4-5d5b-10b5-245b-74317cc049aa

To add or remove devices from passthough, shut down the VM, edit the command to add or remove the <pci-id#>, rerun the edited command, and then restart the VM. For example, to remove all devices from the VM in the above example, shutdown the VM and run the following command:

xe vm-param-set  other-config:pci=  uuid=0370e4b4-5d5b-10b5-245b-74317cc049aa

I hope that the above is of help to anyone who wants to give XCP a try.

[heiko_s]

@GizmoChicken: Thanks for sharing your experience with XCP ! Good to see that you ventured into new territory.

I never gave XCP a try as I heard that some things are easier to accomplish while others are more difficult. Once I solved the hardware part (graphics card), I felt that Xen is rather straight forward.

With regard to the Linux domU VGA passthrough, I suppose you had the Linux guest configured as HVM guest. I'm wondering how a PV guest would work, and if it could use the proprietary graphics drivers, just like dom0? Actually I never gave it much thought, as I assumed that dom0 was nothing but a privileged PV domU. Did you try a Linux domU as a PV guest?