Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 28

Thread: Xen crashes entire server when creating a virtual machine

  1. #11
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    I copied the xlexample.pvlinux file and modified it. Here's my modified config file:

    Code:
    # =====================================================================
    # Example PV Linux guest configuration
    # =====================================================================
    #
    # This is a fairly minimal example of what is required for a
    # Paravirtualised Linux guest. For a more complete guide see xl.cfg(5)
    
    # Guest name
    name = "Dom4Server"
    arch = "i686"
    
    # 128-bit UUID for the domain as a hexadecimal number.
    # Use "uuidgen" to generate one if required.
    # The default behavior is to generate a new UUID each time the guest is started.
    #uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
    
    # Kernel image to boot
    kernel = "/var/lib/xen/images/ubuntu-netboot/trusty14LTS/vmlinuz"
    ramdisk = "/var/lib/xen/images/ubuntu-netboot/trusty14LTS/initrd.gz"
    
    # Ramdisk (optional)
    #ramdisk = "/boot/initrd.gz"
    
    # Kernel command line options
    extra = "root=/dev/xvda1"
    
    # Initial memory allocation (MB)
    memory = 2048
    
    # Maximum memory (MB)
    # If this is greater than `memory' then the slack will start ballooned
    # (this assumes guest kernel support for ballooning)
    #maxmem = 512
    
    # Number of VCPUS
    vcpus = 1
    
    # Network devices
    # A list of 'vifspec' entries as described in
    # docs/misc/xl-network-configuration.markdown
    vif = [ '' ]
    
    # Disk Devices
    # A list of `diskspec' entries as described in
    # docs/misc/xl-disk-configuration.txt
    disk = [ '/etc/xen/dom4server.img,raw,xvda,rw' ]
    Unless I'm mistaken, this should be a PV config..

  2. #12
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    I think arch in an XL type of cinfig file detects the arch from whatever the installed OS is. so you have to be careful on what is installed. So not a vailid option as you put there. That is if you had an XML styled config file.

    Other pv options: (from http://xenbits.xen.org/docs/unstable...ecific-Options )
    Paravirtualised (PV) Guest Specific Options

    The following options apply only to Paravirtual guests.

    bootloader="PROGRAM"
    Run PROGRAM to find the kernel image and ramdisk to use. Normally PROGRAM would be pygrub, which is an emulation of grub/grub2/syslinux. Either kernel or bootloader must be specified for PV guests.

    bootloader_args=[ "ARG", "ARG", ...]
    Append ARGs to the arguments to the bootloader program. Alternatively if the argument is a simple string then it will be split into words at whitespace (this second option is deprecated).

    e820_host=BOOLEAN
    Selects whether to expose the host e820 (memory map) to the guest via the virtual e820. When this option is false (0) the guest pseudo-physical address space consists of a single contiguous RAM region. When this option is specified the virtual e820 instead reflects the host e820 and contains the same PCI holes. The total amount of RAM represented by the memory map is always the same, this option configures only how it is laid out.

    Exposing the host e820 to the guest gives the guest kernel the opportunity to set aside the required part of its pseudo-physical address space in order to provide address space to map passedthrough PCI devices. It is guest Operating System dependent whether this option is required, specifically it is required when using a mainline Linux ("pvops") kernel. This option defaults to true (1) if any PCI passthrough devices are configured and false (0) otherwise. If you do not configure any passthrough devices at domain creation time but expect to hotplug devices later then you should set this option. Conversely if your particular guest kernel does not require this behaviour then it is safe to allow this to be enabled but you may wish to disable it anyway.

    pvh=BOOLEAN
    Selects whether to run this PV guest in an HVM container. Default is 0.
    But I didn't see anyhting obvious that would be loading HVM drivers.
    Last edited by MAFoElffen; July 7th, 2016 at 04:59 AM.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  3. #13
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    Hmm, do you still have any suggestions about how to continue with this?

    Also, it's weird that sometimes the DomU installation progressed quite far and it downloaded setup files from internet, and sometimes it crashed pretty early in the setup..

  4. #14
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    Let me sleep on this. I have 2 servers to upgrade here. Once I do, then I'll take my server that is similar to your server board and install Xen on the hardware. That way maybe i can recreate your problem.

    But like I said, I need to get those 2 back up first... and I'm waiting on one part to get here for the second of the two.

    Are you creating the DomU from the commandline? If so, then post that, so I can create the same.

    EDIT:
    Status-- One server back online. Next server still waiting on it's part, which should get here the 22nd.
    Last edited by MAFoElffen; July 19th, 2016 at 02:49 PM.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  5. #15
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    Yes, I created the DomU from command line with this command

    Code:
    sudo xl create -c /etc/xen/dom4server.cfg
    where the file is the config file listed earlier.

  6. #16
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    Looks there more weird stuff going on. When I was trying to copy stuff to my samba share with just Dom0 running, I got errors like these:


    Code:
    heinš 19 21:58:29 ruoska kernel: BUG: unable to handle kernel paging request at ffff88007ff80000heinš 19 21:58:29 ruoska kernel: IP: [<ffffffff813fe01e>] do_csum+0x7e/0x170
    heinš 19 21:58:29 ruoska kernel: PGD 1e0b067 PUD 1f9c46067 PMD 1f9a46067 PTE 0
    heinš 19 21:58:29 ruoska kernel: Oops: 0000 [#1] SMP 
    heinš 19 21:58:29 ruoska kernel: Modules linked in: xen_gntdev xen_evtchn xenfs xen_privcmd zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) ppdev bridge stp llc serio_raw edac_mce_amd shpchp k8temp edac_core parport_pc parport 8250_fintek i2c_nforce2 mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi e1000 sata_mv sata_nv pata_amd fjes
    heinš 19 21:58:29 ruoska kernel: CPU: 0 PID: 2893 Comm: smbd Tainted: P           O    4.4.0-31-generic #50-Ubuntu
    heinš 19 21:58:29 ruoska kernel: Hardware name: TYAN Computer Corp. S2895/S2895, BIOS 2004Q3 11/18/2008
    heinš 19 21:58:29 ruoska kernel: task: ffff88021f581900 ti: ffff8800ab280000 task.ti: ffff8800ab280000
    heinš 19 21:58:29 ruoska kernel: RIP: e030:[<ffffffff813fe01e>]  [<ffffffff813fe01e>] do_csum+0x7e/0x170
    heinš 19 21:58:29 ruoska kernel: RSP: e02b:ffff8800ab283a90  EFLAGS: 00010286
    heinš 19 21:58:29 ruoska kernel: RAX: 8c121bd87fe9fd90 RBX: 0000000000000000 RCX: 0000000000000000
    heinš 19 21:58:29 ruoska kernel: RDX: ffff880096d446a8 RSI: 000000002214a7fa RDI: ffff88007ff7ffe8
    heinš 19 21:58:29 ruoska kernel: RBP: ffff8800ab283a90 R08: 0000000000000000 R09: 00000000044294ff
    heinš 19 21:58:29 ruoska kernel: R10: 0000000000000000 R11: 0000000254289cfa R12: 000000002214a800
    heinš 19 21:58:29 ruoska kernel: R13: 000000002214a800 R14: ffff880215b3a608 R15: 0000000000000000
    heinš 19 21:58:29 ruoska kernel: FS:  00007f51201258c0(0000) GS:ffff880229e00000(0000) knlGS:0000000000000000
    heinš 19 21:58:29 ruoska kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
    heinš 19 21:58:29 ruoska kernel: CR2: ffff88007ff80000 CR3: 00000000aecf1000 CR4: 0000000000000660
    heinš 19 21:58:29 ruoska kernel: Stack:
    heinš 19 21:58:29 ruoska kernel:  ffff8800ab283aa8 ffffffff813fe151 ffff880215c2d900 ffff8800ab283ab8
    heinš 19 21:58:29 ruoska kernel:  ffffffff8170d869 ffff8800ab283b28 ffffffff8170fb75 0000000000000000
    heinš 19 21:58:29 ruoska kernel:  0000000000000000 000000018010000c ffff880215c2d900 2214a800d15796ef
    heinš 19 21:58:29 ruoska kernel: Call Trace:
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff813fe151>] csum_partial+0x11/0x20
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8170d869>] csum_partial_ext+0x9/0x10
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8170fb75>] __skb_checksum+0x65/0x2d0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8170fe15>] skb_checksum+0x35/0x50
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8170d860>] ? skb_panic+0x70/0x70
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8170d0a0>] ? reqsk_fastopen_remove+0x170/0x170
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff81715b42>] __skb_checksum_complete+0x22/0xd0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8177b30f>] tcp_rcv_established+0x18f/0x780
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff81786165>] tcp_v4_do_rcv+0x145/0x200
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8170a5c5>] release_sock+0x95/0x160
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8176f73c>] tcp_recvmsg+0x3fc/0xbe0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8179d87e>] inet_recvmsg+0x7e/0xb0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff81705eab>] sock_recvmsg+0x3b/0x50
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff81705f52>] sock_read_iter+0x92/0xe0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8120ca3c>] do_iter_readv_writev+0x6c/0xa0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8120d5bf>] do_readv_writev+0x18f/0x230
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8120d696>] vfs_readv+0x36/0x50
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8120e329>] SyS_readv+0x59/0xf0
    heinš 19 21:58:29 ruoska kernel:  [<ffffffff8182db32>] entry_SYSCALL_64_fastpath+0x16/0x71
    heinš 19 21:58:29 ruoska kernel: Code: d1 c1 ea 04 41 d1 e9 85 d2 0f 84 06 01 00 00 83 ea 01 45 31 c0 48 83 c2 01 48 c1 e2 06 48 01 fa 48 03 07 48 13 47 08 48 13 47 10 <48> 13 47 18 48 13 47 20 48 13 47 28 48 13 47 30 48 13 47 38 4c 
    heinš 19 21:58:29 ruoska kernel: RIP  [<ffffffff813fe01e>] do_csum+0x7e/0x170
    heinš 19 21:58:29 ruoska kernel:  RSP <ffff8800ab283a90>
    heinš 19 21:58:29 ruoska kernel: CR2: ffff88007ff80000
    
    
    heinš 19 21:58:29 ruoska kernel: ---[ end trace 0fb3718651e29e0f ]---



    Code:
    heinš 19 22:06:24 ruoska kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [smbd:2957]heinš 19 22:06:24 ruoska kernel: Modules linked in: xen_gntdev xen_evtchn xenfs xen_privcmd zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) ppdev bridge stp llc serio_raw edac_mce_amd shpchp k8temp edac_core parport_pc parport 8250_fintek i2c_nforce2 mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi e1000 sata_mv sata_nv pata_amd fjes
    heinš 19 22:06:24 ruoska kernel: CPU: 1 PID: 2957 Comm: smbd Tainted: P      D    O    4.4.0-31-generic #50-Ubuntu
    heinš 19 22:06:24 ruoska kernel: Hardware name: TYAN Computer Corp. S2895/S2895, BIOS 2004Q3 11/18/2008
    heinš 19 22:06:24 ruoska kernel: task: ffff8800cd21cb00 ti: ffff880185e5c000 task.ti: ffff880185e5c000
    heinš 19 22:06:24 ruoska kernel: RIP: e030:[<ffffffff81723091>]  [<ffffffff81723091>] napi_gro_flush+0x21/0x80
    heinš 19 22:06:24 ruoska kernel: RSP: e02b:ffff880229e83db8  EFLAGS: 00000286
    heinš 19 22:06:24 ruoska kernel: RAX: ffff8801fc6a6000 RBX: ffff88021dcc8af0 RCX: ffff88022223f000
    heinš 19 22:06:24 ruoska kernel: RDX: ffff8801fc6a6000 RSI: 0000000000000000 RDI: ffff8801fc6a6000
    heinš 19 22:06:24 ruoska kernel: RBP: ffff880229e83dd0 R08: 0000000000000002 R09: 0000000000000000
    heinš 19 22:06:24 ruoska kernel: R10: ffff8801fa5d1ec0 R11: 000000001917ff00 R12: ffff88021dcc8af0
    heinš 19 22:06:24 ruoska kernel: R13: 0000000000000000 R14: 0000000000000002 R15: ffff88021dcc8af0
    heinš 19 22:06:24 ruoska kernel: FS:  00007f51201258c0(0000) GS:ffff880229e80000(0000) knlGS:0000000000000000
    heinš 19 22:06:24 ruoska kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
    heinš 19 22:06:24 ruoska kernel: CR2: 00007f76983cff90 CR3: 0000000196518000 CR4: 0000000000000660
    heinš 19 22:06:24 ruoska kernel: Stack:
    heinš 19 22:06:24 ruoska kernel:  ffff88021dcc8af0 0000000000004e20 0000000000000003 ffff880229e83df0
    heinš 19 22:06:24 ruoska kernel:  ffffffff81723158 ffff88021dcc8af0 0000000000004e20 ffff880229e83ea0
    heinš 19 22:06:24 ruoska kernel:  ffffffffc004e831 000000000000001c ffff880229e83e20 0000000000000000
    heinš 19 22:06:24 ruoska kernel: Call Trace:
    heinš 19 22:06:24 ruoska kernel:  <IRQ> 
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81723158>] napi_complete_done+0x68/0xb0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffffc004e831>] e1000_clean+0x371/0x8c0 [e1000]
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff817233be>] net_rx_action+0x21e/0x360
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81085b51>] __do_softirq+0x101/0x290
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81085e53>] irq_exit+0xa3/0xb0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff814d1725>] xen_evtchn_do_upcall+0x35/0x40
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8182f8ae>] xen_do_hypervisor_callback+0x1e/0x40
    heinš 19 22:06:24 ruoska kernel:  <EOI> 
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8102345d>] ? xen_force_evtchn_callback+0xd/0x10
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81023c82>] ? check_events+0x12/0x20
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81023c6f>] ? xen_restore_fl_direct_reloc+0x4/0x4
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81085590>] ? do_softirq.part.19+0x30/0x40
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8108561d>] ? __local_bh_enable_ip+0x7d/0x80
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8182d65e>] ? _raw_spin_unlock_bh+0x1e/0x20
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8170a641>] ? release_sock+0x111/0x160
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8176f73c>] ? tcp_recvmsg+0x3fc/0xbe0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8179d87e>] ? inet_recvmsg+0x7e/0xb0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81705eab>] ? sock_recvmsg+0x3b/0x50
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff81705f52>] ? sock_read_iter+0x92/0xe0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8120ca3c>] ? do_iter_readv_writev+0x6c/0xa0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8120d5bf>] ? do_readv_writev+0x18f/0x230
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8182fed9>] ? error_exit+0x9/0x20
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8120d696>] ? vfs_readv+0x36/0x50
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8120e329>] ? SyS_readv+0x59/0xf0
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8182db32>] ? entry_SYSCALL_64_fastpath+0x16/0x71
    heinš 19 22:06:24 ruoska kernel:  [<ffffffff8102345d>] ? xen_force_evtchn_callback+0xd/0x10
    
    
    heinš 19 22:06:24 ruoska kernel: Code: eb df 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 31 d2 48 89 e5 41 55 41 54 49 89 fc 53 48 8b 7f 38 41 89 f5 48 85 ff 75 05 eb 44 <48> 89 c7 48 8b 07 48 89 57 08 48 89 fa 48 85 c0 75 ee eb 03 48
    Last edited by Mikael_Niemel; July 19th, 2016 at 08:55 PM.

  7. #17
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    Looks like transfer went just fine with plain Ubuntu kernel.. nothing out of ordinary on dmesg.

    I have no idea why this had not happened earlier, maybe it's because it was the first time I had transferred very large amounts of data with this installation?
    Last edited by Mikael_Niemel; July 19th, 2016 at 09:51 PM.

  8. #18
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    Mikael-

    I've seen the error in your your last post before on the Xen Development Mailing List, There, it was corrected and went away by his next distro kernel update. When I chased another Debian Bug report down with the same eror under Xen... it was traced to an XSave option in the code of the kernel (in the kernel config file used when compiling), that affects some kernel/cpu combinations.. What that translates to, is that Dom0, leaves a saved page in memory, when is was actually through with it. When something tries to reuse that page (generates a page request), it gets confused whether it can or not (results in bad page request).

    Try to add these options into /etc/default/grub, in line GRUB_CMDLINE_XEN:
    Code:
    noxsave=0 noxsaveopt=0
    ...which will tell the kernel not to save those unused pages in memory.

    Another "fix" might be to update your system, which would pick up a newer kernel. There has been another released kernel update since you first got this error... and see how it does with a newer kernel.
    Last edited by MAFoElffen; July 20th, 2016 at 03:04 PM.
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

  9. #19
    Join Date
    May 2016
    Beans
    37

    Re: Xen crashes entire server when creating a virtual machine

    I tried adding those options and getting updates with apt-get, but apparently kernel version was already 4.4.0-31-generic, which is the newest I think.

    Anyways, it didn't work. It still crashed, and dumped errors to console before rebooting automatically within a second or so. This time there indeed were no logs saved (journalctl didn't show anything for several minutes before the crash), so I don't know what caused it. I think there was something about "fatal exception in an interrupt - not syncing" in the last line of the error dump that I saw by glimpse.

    Apparently my board's BIOS has console redirection to serial port, so I suppose I could use that to save the error messages somewhere.. didn't find any BIOS logs, though.

    EDIT: I tried again, and got the same error to journalctl as in previous page.
    Last edited by Mikael_Niemel; July 20th, 2016 at 06:14 PM.

  10. #20
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: Xen crashes entire server when creating a virtual machine

    Okay, well... Just got an email that my heat sink for the second cpu in my second server got held up in customs and shouldn't be here until the 28th. Since that got held up, I'll push the first server online, to pull that older server with the Tyan board down to config like you have yours.

    As a test, I'm thinking I should install and test on kernel 4.4.0-28, before upgrading to 4.4.0-31... Reason? I was thinking about you today = I tried installing Xen directly onto that new server, while it was at 4.4.0-31 ... and it crashed so bad it would not boot. I ended up reinstalling a fresh install onto it!!!
    Concurrent coexistance of Windows, Linux and Unix...
    Ubuntu user # 33563, Linux user # 533637
    Sticky: [all variants] Graphics Resolution- Upgrade /Blank Screen after reboot

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •