Page 1 of 3 123 LastLast
Results 1 to 10 of 28

Thread: Xen crashes entire server when creating a virtual machine

  1. #1
    Join Date
    May 2016
    Beans
    37

    Xen crashes entire server when creating a virtual machine

    Looks like my Xen problems aren't over. When I try to create a Xen virtual machine, I get a bunch of ATA related errors, and after that the entire server crashes and/or reboots.

    Here's the first message, I wasn't even able to save any logs, so I had to manually take a pic of them before the server rebooted..

    Code:
    ata9: illegal qc_active transition (0001c000->0000bfff)
    After that I got a bunch of

    Code:
    ata<x>.00: failed command: READ FPDMA QUEUED
    Amongst the errors there also was a

    Code:
    ata<x>.00: PCI err cause 0x9ca634e3


    Anyways, I'm trying to use a ZFS volume in Dom0 for VM storage, should it be okay for Xen?
    Last edited by Mikael_Niemel; June 25th, 2016 at 06:47 PM.

  2. #2
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    You have said Xen and ZFS... But I see an incomplte picture.

    What is the OS and version of, that you have Xen running on? What is your version of Xen. Are you suing the newer kernel module or FUSE for ZFS? Is this error on your host or guest? If on a guest, what is the OS and verson of the guest?

    Here is why I ask those-- The times I've seen that error was either on iron, when SATA first cam out, and when the nwer SATA drives came out... and controllers where having issues reading the nwer drives. The other times I've seen this error under a hypervisor, is when there was a PCI passthrough and there was a problem with the passthrough...

    So please explain the circumstances surrounding your system and the errors.

    EDIT--
    Depending on how, which implementation of the ZFS interface, etc... I have no issues with ZFS. I've used ZFS for over about 9-10 years. I've run Ubuntu on ZFS on metal with no issues. I've run Ubuntu on ZFS in KVM with no issues. I can't say I've run Ubuntu on ZFS in any of my Xen Server, but I can say Solaris on ZFS in Xen has no issues on any on my Xen Servers.
    Last edited by MAFoElffen; June 26th, 2016 at 07:41 AM.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  3. #3
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    I'm using Ubuntu 16.04 for host, kernel version is 4.4.0-24-generic. I installed ZFS just by doing
    Code:
    sudo apt install zfs
    , if I understood correctly, that means the new kernel module. Without Xen DomUs running, ZFS has been working just fine in Ubuntu.

    Also, this error is on host, not on guest, which is Ubuntu 14.04 LTS. No errors are seen on guest before the system crashes, and I haven't set up PCI passthrough either. Xen version is 4.6.0.

    I'm using a AOC-SAT-MV8 controller to connect my S-ATA drives for ZFS array, but my OS disk (a SSD) is on a MB S-ATA port. I'll check if this happens on a cheapo Sil3124 controller, though I have had other problems with it.
    Last edited by Mikael_Niemel; June 26th, 2016 at 01:17 PM.

  4. #4
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    You know, funny you should mention that (Xen and S-ATA). I've seen a few problems lately from users using Xen with S-ATA as being the culprits. I think the most recent was in this Virtualization Section, started about week ago. So a coincidence that you mention that.

    Look at this thread, page 2. posts 18 & 19: http://ubuntuforums.org/showthread.php?t=2326179 It is now solved (was related to S-ATA) and I'm now workig out a few details on a side question with him. It was not related to ZFS. It was just Xen and S-ATA.

    I tested the ZFS modules during the last 2 dev cycles and it is so much better than the previous fuse implementation. It tested out fine for me. I don't suspect ZFS itself, but I didn't test it with S-ATA and Xen. Admittedly, I used LVM on hardware RAID on my KVM and Xen Servers.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  5. #5
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    Quote Originally Posted by MAFoElffen View Post
    You know, funny you should mention that (Xen and S-ATA). I've seen a few problems lately from users using Xen with S-ATA as being the culprits. I think the most recent was in this Virtualization Section, started about week ago. So a coincidence that you mention that.

    Look at this thread, page 2. posts 18 & 19: http://ubuntuforums.org/showthread.php?t=2326179 It is now solved (was related to S-ATA) and I'm now workig out a few details on a side question with him. It was not related to ZFS. It was just Xen and S-ATA.

    I tested the ZFS modules during the last 2 dev cycles and it is so much better than the previous fuse implementation. It tested out fine for me. I don't suspect ZFS itself, but I didn't test it with S-ATA and Xen. Admittedly, I used LVM on hardware RAID on my KVM and Xen Servers.
    Umm, that linked thread is my previous thread, which was about Xen not even booting to Dom0 (I started a new thread as this seemed like a new issue). It turned out that it didn't recognize old IDE disks (and the Ubuntu instance was on a IDE HDD), but it booted fine to Dom0 when I re-installed the instance to a S-ATA disk.

  6. #6
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release
    LOL, sorry... So it is really still having troubles along the same line then.

    I've been looking at that limiting to 32 bit guests with the newer XenLight Toolset for you. That has me been driving me batty. While a DomU is running or active (but stopped), I can see the xml and json for it. Before that,, well, it has to be stored somewhere, but I haven't found where yet.

    In the xml, it's linked to the os type equals i686, where it you did
    Code:
    sudo virsh edit DomainName
    It would then change that... but had family things this weekend, so not a lot of time. In the json, I can see it affects with version of the kernel it is using to virtualize the Dom. But since I can't figure out where it is storing the config itself, in it's own internal format, I can't see what the change is, to be able to change the default xl.conf. But still chasing that.

    But you are saying you are still getting I/O errors. You said journalctl doesn't work for you? With dmesg, can you tell if is it getting the errors on the physical disk, or on the filesystem? Is there sensitive data in it where that would preclude posting a filtered result to ubuntu.pastebin.com? Like pm me the link, so the world does not have to get access to it?

    EDIT#1--
    I didn't ask in the other thread, what is the spec's of you server. Wondering the cpu, memory, etc. and version of Xen.

    Sidenote-- Mine is Xen version 4.6 on 16.04. I am not without any errors. It seems to want to lose the cursor on a Windows 2016 preview 5 DomU in some viewers, if I switch between windows (it loses focus, then gets focus again).

    EDT#2--
    I told you I was using LVM. Are you using pools for your DomU Domain/Guest or creating images files? One way of creating DomU's is to create and LVM Volume Group, then install the DomU to that Volume Group. For that to happen, the Volume Group has to pre-exist. I was wondoering if you were trying to to the same logical method using ZFS (something I have never tried, but I don't see why it wouldn't work in the same manner).

    If to it's own ZFS pools, then may be more of a logic error, were it is trying to write to something it can't fully find? Just thinking out loud.

    So have you been able to successfully create any DomU on your new Xen install yet?

    EDIT#3--
    So 3 years ago, when it was still Fuse ZFSonLinux... Users on the Xen User Mail List said the ZFS pool had to pre-exist. There was a problem with creation directly one ZFS, so they had this work-around:
    The procedure I followed is:
    0) set options in /etc/xen-tools/xen-tools.conf
    Code:
            dir = /home/xen/ 
            fs     = ext3 
            image  = sparse
    1) create DomU eg:
    Code:
    xen-create-image --hostname vitrualM --memory=8gb --dhcp --vcpus 2 --pygrub --dist wheezy
    this creates volumes in /home/xen/domains/virtualM/
    Code:
    disk.img 
    swap.img
    2) create zvol, eg:
    Code:
    zfs create -V 100G zpool1/vitrualM
    3) use dd to copy over image to zvol:
    Code:
    dd if=/home/xen/domains/vitrualM/disk.img of=/dev/zvol/zpool1/vitrualM bs=1M
    4) set DomU config to point to new location of image:
    Code:
    'phy:/dev/zvol/zpool1/vitrualM,xvda2,w',
    I haven't confirmed this, but ... I'm suspecting that now-a-days, with the ZFS support now in the newer kernel, that if you declared the new zvol, then pointed the create statement to that zvol... that it may write it directly to that zvol. If not, then there is still that old work-around.

    Using the above example, with LVM you would use
    Code:
    xen-create-image --hostname vitrualM --memory=8gb --dhcp --vcpus 2 --pygrub --dist wheezy --image full --lvm my--vg--virtualM
    I know that when using LVM, that you pre-define, and that these options are important:
    Code:
     −−image   Specify whether to create "sparse" or "full" disk images. Full images are mandatory when using LVM, so this setting
     −−dir   Specify where the output images should go. Subdirectories will be created for each guest. If you do not wish to use loopback images specify −−lvm or −−evms.  (These three options are mutually exclusive.)
     −−lvm   Specify the volume group to save images within. If you do not wish to use LVM specify −−dir or −−evms. (These three options are mutually exclusive.)
     −−evms   Specify the container to save images within, i.e. ’−−evms lvm2/mycontainer’.  If you do not wish to use EVMS specify −−dir or −−lvm.  (These three options are mutually exclusive.)
    So, for a pre-defined zvol, I see it as possibly (guess) being something like
    Code:
    xen-create-image --hostname vitrualM --memory=8gb --dhcp --vcpus 2 --pygrub --dist wheezy --image full --dir /Path_To/New_ZVol/vitrualM
    But that might also be via evms. If creating new domains from config files, then are you directly pointing the config to a pre-deined zvol?
    Last edited by MAFoElffen; June 27th, 2016 at 06:06 PM.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  7. #7
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    I have been a bit busy for some days, but I'm now back again..

    Quote Originally Posted by MAFoElffen View Post
    LOL, sorry... So it is really still having troubles along the same line then.

    I've been looking at that limiting to 32 bit guests with the newer XenLight Toolset for you. That has me been driving me batty. While a DomU is running or active (but stopped), I can see the xml and json for it. Before that,, well, it has to be stored somewhere, but I haven't found where yet.

    In the xml, it's linked to the os type equals i686, where it you did
    Code:
    sudo virsh edit DomainName
    It would then change that... but had family things this weekend, so not a lot of time. In the json, I can see it affects with version of the kernel it is using to virtualize the Dom. But since I can't figure out where it is storing the config itself, in it's own internal format, I can't see what the change is, to be able to change the default xl.conf. But still chasing that.

    Hmm, it says that command virsh isn't found when I try that..

    Quote Originally Posted by MAFoElffen View Post
    But you are saying you are still getting I/O errors. You said journalctl doesn't work for you? With dmesg, can you tell if is it getting the errors on the physical disk, or on the filesystem? Is there sensitive data in it where that would preclude posting a filtered result to ubuntu.pastebin.com? Like pm me the link, so the world does not have to get access to it?

    EDIT#1--
    I didn't ask in the other thread, what is the spec's of you server. Wondering the cpu, memory, etc. and version of Xen.

    Sidenote-- Mine is Xen version 4.6 on 16.04. I am not without any errors. It seems to want to lose the cursor on a Windows 2016 preview 5 DomU in some viewers, if I switch between windows (it loses focus, then gets focus again).

    Thing is that when the errors appear, the entire server crashes or reboots before I can save the errors anywhere. The only way I was able to get some dmesg logs out was by taking a picture of the errors.

    Anyways, my Xen is version 4.6 on ubuntu 16.04 as well. My server motherboard is a Tyan Thunder K8WE with two Opteron 275 CPUs and 24Gb memory.




    Quote Originally Posted by MAFoElffen View Post
    EDT#2--
    I told you I was using LVM. Are you using pools for your DomU Domain/Guest or creating images files? One way of creating DomU's is to create and LVM Volume Group, then install the DomU to that Volume Group. For that to happen, the Volume Group has to pre-exist. I was wondoering if you were trying to to the same logical method using ZFS (something I have never tried, but I don't see why it wouldn't work in the same manner).

    If to it's own ZFS pools, then may be more of a logic error, were it is trying to write to something it can't fully find? Just thinking out loud.

    So have you been able to successfully create any DomU on your new Xen install yet?
    I have been trying to use ZFS volumes created to the single ZFS pool I have in the system. I basically created a ZFS volume to my pool by

    Code:
    zfs create -V 20gb data/testserver
    Then I set that volume as the disk to the Xen config file, it's path was "/dev/zvol/data/testserver".


    Anyways, I'll try to use file based storage for DomU and see if that works.. Now the install of guest system is going on with no errors.. yet.

    EDIT: No luck, it still crashes even when using file based storage. Next I'll try disabling ZFS and using a file on my OS disk..

    EDIT2: Nope, it still doesn't work, so it shouldn't be about ZFS. So looks like I'm not able to successfully create any DomU..
    Last edited by Mikael_Niemel; July 4th, 2016 at 05:09 PM.

  8. #8
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    Quote Originally Posted by Mikael_Niemel View Post
    Anyways, my Xen is version 4.6 on ubuntu 16.04 as well. My server motherboard is a Tyan Thunder K8WE with two Opteron 275 CPUs and 24Gb memory.
    LOL! Very small world! I have an old Test Server that is Tyan K8SE S2892G3NR. I have a spare board that box, that is a ASUS K8N-DRE. So Both are a few moths older than yours, but not by much. Both my Boards use the same Opteron Series 200 processors as yours. So yes, I am familiar with your hardware. I think the main different between the series I have and yours, was that your did SLI.

    Now I understand some of your configuration considerations.On those boards, we have socket 940 and Opteron 200 series. AMD-v didn't come out until later with "Paifica" and Socket F processors, like the Opteron 2000 series.. So no iommu on those.

    No no idea? That board has BIOS server logs a irght? No messages there? Then, Xen server does journalctl journaling... but you said yours does not right? Cause if you turned on persistent journaling, then you would have the system log of that crash...

    Just thinking out loud. I guess my old test server is similar to yours to slap together for a Xen test with ZFS... I need to get a few other things done before I could do that. But if need be, at least it's there and available.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  9. #9
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    I checked again logs with journalctl, and now it actually had logs in it. IIRC it didn't initially work, but I'm not sure if I have just been brainfarting or something..

    Anyways, here's what I got from it this time, I'm really sorry that it took so long for me to get some useful data for solving this..

    Code:
    heinä 04 18:52:54 ruoska kernel: BUG: unable to handle kernel paging request at ffffc910436df210heinä 04 18:52:54 ruoska kernel: IP: [<ffffffffc034a441>] xenvif_kthread_guest_rx+0x4e1/0xa20 [xen_netback]
    heinä 04 18:52:54 ruoska kernel: PGD 1f6fac067 PUD 0 
    heinä 04 18:52:54 ruoska kernel: Oops: 0000 [#1] SMP 
    heinä 04 18:52:54 ruoska kernel: Modules linked in: xt_physdev br_netfilter iptable_filter ip_tables x_tables xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd ppdev bridge stp llc input_leds serio_raw k8temp edac_mce_amd edac_core shpchp parport_pc 8250_fintek parport i2c_nforce2 mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi e1000 sata_mv sata_nv pata_amd fjes
    heinä 04 18:52:54 ruoska kernel: CPU: 3 PID: 1788 Comm: vif1.0-q0-guest Not tainted 4.4.0-28-generic #47-Ubuntu
    heinä 04 18:52:54 ruoska kernel: Hardware name: TYAN Computer Corp. S2895/S2895, BIOS 2004Q3 11/18/2008
    heinä 04 18:52:54 ruoska kernel: task: ffff88022271cb00 ti: ffff88021d91c000 task.ti: ffff88021d91c000
    heinä 04 18:52:54 ruoska kernel: RIP: e030:[<ffffffffc034a441>]  [<ffffffffc034a441>] xenvif_kthread_guest_rx+0x4e1/0xa20 [xen_netback]
    heinä 04 18:52:54 ruoska kernel: RSP: e02b:ffff88021d91fdf8  EFLAGS: 00010286
    heinä 04 18:52:54 ruoska kernel: RAX: ffffc910436aa000 RBX: ffffc9004372c9d8 RCX: 0000000000000001
    heinä 04 18:52:54 ruoska kernel: RDX: 00000000ffff8800 RSI: 0000000000000000 RDI: 0000000000000005
    heinä 04 18:52:54 ruoska kernel: RBP: ffff88021d91fec0 R08: 000000000000062a R09: ffff880005a94840
    heinä 04 18:52:54 ruoska kernel: R10: ffffc90043722000 R11: ffffc9004372c9d8 R12: 0000000000000000
    heinä 04 18:52:54 ruoska kernel: R13: 0000000000000000 R14: ffffc90043722000 R15: ffff88021d91fe3c
    heinä 04 18:52:54 ruoska kernel: FS:  00007f3a077ed700(0000) GS:ffff880229f80000(0000) knlGS:0000000000000000
    heinä 04 18:52:54 ruoska kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
    heinä 04 18:52:54 ruoska kernel: CR2: ffffc910436df210 CR3: 00000000cce0c000 CR4: 0000000000000660
    heinä 04 18:52:54 ruoska kernel: Stack:
    heinä 04 18:52:54 ruoska kernel:  ffff88021d91fe3c ffff88022271cb00 ffff880229e96d00 ffffc90043757208
    heinä 04 18:52:54 ruoska kernel:  ffff880005a94840 ffffc9004372ca00 ffffc9004372c9d8 ffff880200000003
    heinä 04 18:52:54 ruoska kernel:  0000000000000000 ffff88021d91fe40 ffff88021d91fe40 ffff88021ee35e00
    heinä 04 18:52:54 ruoska kernel: Call Trace:
    heinä 04 18:52:54 ruoska kernel:  [<ffffffff81827a4f>] ? ret_from_fork+0x3f/0x70
    heinä 04 18:52:54 ruoska kernel:  [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0
    heinä 04 18:52:54 ruoska kernel: Code: 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 48 89 42 08 48 89 10 8b 45 b4 4d 8b 4e 20 48 89 c2 48 c1 e0 04 41 8b 71 28 4c 01 f0 <8b> 88 10 52 03 00 0f a3 ce 73 5a 41 8b 8e c0 a9 00 00 49 8b be 
    heinä 04 18:52:54 ruoska kernel: RIP  [<ffffffffc034a441>] xenvif_kthread_guest_rx+0x4e1/0xa20 [xen_netback]
    heinä 04 18:52:54 ruoska kernel:  RSP <ffff88021d91fdf8>
    heinä 04 18:52:54 ruoska kernel: CR2: ffffc910436df210
    heinä 04 18:52:54 ruoska kernel: ---[ end trace c3c7b08cf215a43c ]---
    heinä 04 18:52:54 ruoska kernel: BUG: unable to handle kernel NULL pointer dereference at           (null)
    heinä 04 18:52:54 ruoska kernel: IP: [<ffffffffc03490ea>] xenvif_rx_queue_tail+0x3a/0xa0 [xen_netback]
    Last edited by Mikael_Niemel; July 4th, 2016 at 05:41 PM.

  10. #10
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    You know, I'm really surprised that the Xen Kernel starts. The first thing that the requirements says is to test for IOMMU and that virtualization is turned on, which I know those CPU's where not capable of. So lets see what you have and what it is capable of... You would not be ablle to run an HVM guest, which leaves running a PV host.
    PV

    Paravirtualization (PV) is an efficient and lightweight virtualization technique originally introduced by Xen Project, later adopted by other virtualization platforms. PV does not require virtualization extensions from the host CPU. However, paravirtualized guests require a PV-enabled kernel and PV drivers, so the guests are aware of the hypervisor and can run efficiently without emulation or virtual emulated hardware. PV-enabled kernels exist for Linux, NetBSD, FreeBSD and OpenSolaris. Linux kernels have been PV-enabled from 2.6.24 using the Linux pvops framework. In practice this means that PV will work with most Linux distributions (with the exception of very old versions of distros).
    I looked at the error and the specific dump had to do with brctl virtualizing networking, which might have to do with vt-d capabilities being not there. Which, was the module load on the Xen backend... on virtulaization modules...Which isn't going to be possible on a non-iommu cpu... but I don't know that bridge itself is HVM. Bridge can still happen with PV. However, more complex virt networking cannot.

    I found the error: https://lists.xenproject.org/archive.../msg00453.html
    So, being limited to a fully paravirtulized DomU, did you, by mistake, use an HVM templete, instead of an PV template?

    PV fully virtualized configuration options
    Last edited by MAFoElffen; July 5th, 2016 at 03:21 PM.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

Page 1 of 3 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •