Ubuntu 14.04 / Xen 4.4 improved VGA passthrough for AMD cards

[heiko_s]

I did some tests with Ubuntu 14.04 / Xen 4.4 regarding VGA passthrough. Here the hardware:

CPU: Intel i7 3930K (supporting VT-d)
M/B: Asus Sabertooth X79 with latest BIOS
dom0 graphics: AMD / ATI 7770
domU graphics: AMD / ATI 6450
dom0 OS: Ubuntu-Gnome 14.04 Beta2 64bit
domU OS: Windows 7 Professional 64bit
dom0 memory: 8GB
domU memory: 8GB
Toolstack: xl
Device model: device_model_version = 'qemu-xen-traditional'

Results:
Aside from some minor glitches like xtightvncviewer not accepting an empty password and xvnc4viewer crashing more often than not, the new Xen 4.4 with the 3.13 kernel greatly improved the ability to pass through the AMD HD 6450 graphics card. Now xl can start and restart the domU without causing performance issues, or crashing dom0. It is no more necessary to use the xm toolstack for VGA passthrough.

EDIT 25.4.2014: I tested my AMD HD 7770 with Ubuntu 14.04 (non-beta). First boot with VGA passthrough works fine, but rebooting the Windows domU with VGA passthrough doesn't activate the AMD GPU and gives a code 43 in the Windows Device manager.
In short: The guest reboot problem has NOT been solved!

Below is my /etc/xen/guest.cfg file:

builder='hvm'
memory = 8192
name = 'win7'
vcpus=6
pae=1
acpi=1
apic=1
on_xend_stop='shutdown'
vif = [ 'mac=00:16:3e:68:e1:01,bridge=xenbr0' ]
disk = [ '/dev/mapper/guests-win7,raw,hda,rw' , '/home/user/ISOs/Win7.iso,raw,hdc,devtype=cdrom' ]
device_model_version = 'qemu-xen-traditional'
boot='cd'
sdl=0
vnc=1
vncpasswd='password'
stdvga=0
serial='pty'
tsc_mode='default'
viridian=1
usb=1
usbdevice='tablet'
gfx_passthru=0
pci=[ '02:00.0', '02:00.1' ]
localtime=1
pci_power_mgmt=1

In the past I had serious issues getting this AMD 6450 card passed through. I tried various releases of Linux Mint/Ubuntu 12.04/12.10/13.10 with Xen 4.1.2/4.1.3/4.2/4.3 and only Ubuntu 12.04 with an old kernel and Xen 4.1.2 (the original version) worked with the xm toolstack.

More on the experiments and results can be found here (Linux Mint 13/Ubuntu 12.04 with Xen 4.1.2) and here (Ubuntu 14.04 with Xen 4.4).

[heiko_s]

Yesterday I tested my AMD HD 7770 card for VGA passthrough, using the final Ubuntu 14.04 LTS release. It turns out that the guest reboot problem hasn't been solved. I can boot with VGA passthrough only once, after that when I reboot I get only a VNC screen and the Windows Device Manager shows a yellow triangle next to the AMD graphics card. See also my edit above.

What a pain in the neck this is!

[Shocky_Han]

hi,

I follows 'Linux Mint Forum - HOW-TO make dual-boot obsolete using XEN VGA passthrough' and it works.

My setup:
CPU: intel i7-3770
M/B: asrock z77 pro4
dom0 graphics: intel integrated
domU graphics: AMD Radeon HD6750
dom0 OS: Ubuntu Server 14.04 + Gnome Desktop + Virtualization
domU OS: Windows 7 Professional 64bit (with Catalist Control Center, No GPLPV drivers)
dom0 memory: 8GB
domU memory: 8GB
Toolstack: xl
Device model: device_model_version = 'qemu-xen-traditional'

/etc/default/grub:

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M"

/etc/initramfs-tools/modules:

xen-pciback passthrough=1 hide=(01:00.0)(01:00.1)

/etc/xen/win7.cfg:

builder='hvm'
memory = 8192
name = 'windows7'
vcpus=6
pae=1
acpi=1
apic=1
on_xend_stop='shutdown'
on_poweroff='destroy'
on_reboot='restart'
on_crash='restart'
vif = [ 'mac=00:16:3e:68:e1:01,bridge=virbr0' ]
disk = [ 'file:/guest/windows7/disk.img,ioemu:hda,w' , 'file:/guest/windows7/windows7.iso,ioemu:hdc:cdrom,r' ]
device_model_version='qemu-xen-traditional'
boot='dc'
sdl=0
vnc=1
#vncpasswd=''
vnccolsole=0
vncused=0
vncdisplay=2    
stdvga=0
serial='pty'
tsc_mode='default'
viridian=1
usb=1
usbdevice='tablet'
gfx_passthru=0
pci=[ '01:00.0', '01:00.1' ] 
localtime=1
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
hpet=1

Everything goes fine except Windows 7 booting is seemed a lot slower than normal.
There's no guest reboot problem at all.
But guest has a problem for the Standard VGA Graphics Adapter and I wonder it is normal under VGA passthrough or not.

I hope this helps.

Shocky

[heiko_s]

@Shocky_Han: Sorry for the late reply. Thanks for sharing your success story and configuration. Indeed it helps!

The slower than normal Windows boot time is normal with secondary passthrough. In my system - using a SSD for Windows and Linux - it takes Windows 7 around 20 seconds to get to the login. From there on its fast.

I also tested Ubuntu 14.04 with kvm and I managed to run primary passthrough with my AMD Radeon HD 7770 card, which reduced the boot time as there is no secondary (virtual) graphics adapter. I hope to write a how-to when I become more familiar with kvm and time permit. Still, if you managed to run Xen with VGA passthrough, don't bother switching to kvm.

The yellow triangle next to the Standard VGA graphics adapter is normal, as far as I remember. You can disable the Standard VGA adapter via the Windows Device Manager.

If you have a Linux Mint account, please post the Passmark performance under the thread I linked to - see the end of the how-to. If not, can you post it here? (I try to gather some performance data - if you happen to have a copy of Windows installed natively on your machine, i.e. bare metal, it would be great to compare the performance.)

[Shocky_Han]

@heiko_s,
It's a greate idea collecting some performance data.
Because I don't have any LM account, I upload the passmark base line here.
This is the only thing I can save in the PerframnceTest 8.0 Evaluation version.
Also I upload my baseline to web and you can see it from http://www.passmark.com/baselines/V8...id=24178651031
FYI, my overoll rating is 242 and it's very slower than I thought.

[heiko_s]

@heiko_s,
It's a greate idea collecting some performance data.
Because I don't have any LM account, I upload the passmark base line here.
This is the only thing I can save in the PerframnceTest 8.0 Evaluation version.
Also I upload my baseline to web and you can see it from http://www.passmark.com/baselines/V8...id=24178651031
FYI, my overoll rating is 242 and it's very slower than I thought.

Have been a little busy these days. Your result is indeed not impressive. For starters, using a file instead of LVM does slow down disk I/O performance in the VM, which also helps explain the slow Windows boot time.

However, the most disturbing thing is the graphics performance - something is wrong here. Below are my benchmarks for the AMD 7770 running in a VM under KVM. There shouldn't be much of a difference - if any - for VGA passthrough between Xen and KVM.

Passmark 2D safemode KVM AMD7770.jpg Passmark 3D safemode KVM AMD7770.jpg

The 2D benchmark (summary) is 613, the 3D benchmark is 2147. This is far from your results!

[sjhupp]

Oh, wow, glad I found this thread!
I'm trying to pass through an AMD card under Ubuntu 14.04 server running Xen using xl to Windows7.
Currently I have a primary HD5450 to play with, but if I can get something working I'll whack in a HD7770 secondary to use.
My setup has issues with IOMMU though, so appreciate any help.

Should add I have a gigabyte X58 board with a L5520 cpu, and both should support this.

Have enabled virtualisation in the BIOS, and can see:

simon@cavetroll:~$ sudo dmesg | grep -e DMAR -e IOMMU
[ 2.810272] Intel-IOMMU: enabled

Yet Xen doesn't agree:

simon@cavetroll:~$ sudo xl dmesg |grep virtualisation
(XEN) I/O virtualisation disabled

and

simon@cavetroll:~$ cat /proc/cpuinfo| egrep "vmx|svm"
simon@cavetroll:~$

Any suggestions? Feel like I'm failing at the first hurdle!
I did try adding to grub:

# trying to force IOMMU on as hardware does support it
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"


But I don't have pci-back compiled in the kernel I assume (did you do anything special?)

simon@cavetroll:~$ sudo xl pci-assignable-list
libxl: error: libxl_pci.c:376:libxl_device_pci_assignable_list: Looks like pciback driver not loaded

can modprobe it, and make devices pci-assignable, but still won't work.

Do I need to try something with initramfs next to load, and hide devices first?
I had issues booting Ubuntu, and at one stage had the cpu showing support, but I've since changed some AHCI/legacy settings in BIOS. Could that impact?

Open for any suggestions. Would love to get this working.
Thanks.

[Teo_En_Ming]

Dear heiko_s,

Thank you very much for the information which you have provided. I have got Xen VGA Passthrough to work with my Sapphire AMD Radeon HD 6450. Here are the details:

Software Configuration:
Host operating system dom0: Ubuntu 14.04 LTS
Linux kernel: 3.16.1 (self-compiled)
Xen hypervisor: 4.4.1 (self-compiled)
Guest operating system domU: Windows 7 Ultimate HVM domU
Sapphire AMD Radeon HD 6450 Driver DVD: INSTALLED

Hardware Configuration:
Processor: Intel Core i5-4430 CPU @ 3.00GHz (Quad Core)
Motherboard: Asrock B85M Pro4 LGA 1150 Motherboard
Memory: 32 GB DDR3-1600
PCI-E x16 Display Card: Sapphire AMD Radeon HD 6450 1 GB DDR3
VT-x: Enabled in UEFI BIOS
VT-d: Enabled in UEFI BIOS


I have written detailed and comprehensive installation guides for Xen VGA Passthrough newbies.

If you need the installation guides, you can get them here.

(1) Building and Installing Xen 4.x and Linux Kernel 3.x on Ubuntu and Debian Linux

Download link: http://www.mediafire.com/view/ok8ps8..._-_REDUCED.pdf

(2) Xen VGA Passthrough with AMD Display Cards

Download link: http://www.mediafire.com/view/hrquhz...ersion_1.6.pdf

You will need to follow the above installation guides in sequence.


Teo En Ming
Singapore

[sjhupp]

Is there any chance of getting this working with nvidia yet (with a standard GTX750 or similar?)
I did have some success with a HD7770 under EXSi and kvm, but nothing with nvidia yet.

[Teo_En_Ming]

Hi,

Please watch the 6-min Youtube video on Xen VGA Passthrough with AMD Radeon HD 6450 here: http://www.youtube.com/watch?v=C8ZjddNbQIE



Software Configuration:

Host operating system dom0: Ubuntu 14.04 LTS
Linux kernel: 3.16.1 (self-compiled)
Xen hypervisor: 4.4.1 (self-compiled)
Guest operating system domU: Windows 8.1 Enterprise HVM domU
Display card driver installed: Catalyst 14.4

Hardware Configuration:

Processor: Intel Core i5-4430 CPU @ 3.00GHz (Quad Core)
Motherboard: Asrock B85M Pro4 LGA 1150 Motherboard
Memory: 32 GB DDR3-1600
PCI-E x16 Display Card: Sapphire AMD Radeon HD 6450 1 GB DDR3
VT-x: Enabled in UEFI BIOS
VT-d: Enabled in UEFI BIOS

--
Yours sincerely,

Teo En Ming
Singapore