PDA

View Full Version : [ubuntu] 8.04 ->10.04 LVM Raid Recovery



rsteinmetz70112
June 17th, 2010, 10:23 PM
I have upgraded one machine from 8.04 to 10.04. It contains 4 hard drives, and two raid sets configured as LVM with riserfs partitions on them, plus two small partitions for swap and for /boot

Upon reboot I get the message


can't find /dev/mapper/ubuntu-root

And it drops me into a shell.

I have started a desktop live CD and Gparted shows the all of the physical devices as lvm2 devices and fdisk reports the partitions, it also identifies three raid sets -
nvidia_hfdafdba
nvidia_hfdafdba1
nvidia_hfdafdba2

It looks like all of the partitions are there but I can't access them.

I get a message that lvm 2 is not installed but that apt-get can install it.

Does anyone have any idea how to get the raid/lvm combination to be recognized.

ronparent
June 18th, 2010, 12:22 AM
The upgrade may not have3 upgraded lvm to lvm2. Although lvm2 is supposedly backward compatible with lvm, I believe it must be installed to be compatible with the current kernel.

rsteinmetz70112
June 18th, 2010, 06:10 PM
That's seems likely to be what happened.

Using the live cd I can install lvm2 and mdadm and can then activate and recover the configuration. All partitions are there and seem OK.

Now how can you install lvm2 at this point?

ronparent
June 18th, 2010, 07:03 PM
You may have to uninstall and purge lvm prior.

In terminal 'sudo apt-get install lvm2'.

rsteinmetz70112
June 18th, 2010, 10:46 PM
I can't get to terminal on the hard drive. I can istall lvm on the Live CD ram drive but how do I install it on the hard drive, since right now it won't boot?

More information;

If I boot the live CD I can install mdadm and lvm2.

With those installed I can recover the raid device md0 and md1 which contain volume groups remus and data, although one shows as degraded. There is somethind odd about the nvidia mapper devices. They seem to refer to the same physical drives as md0

From the Live CD I can recover the lvm config and that creates the mapper devices for

remus-root
remus-var
remus-tmp
data-files
data-home

These filesystems can be mounted and checked. The data is still there.

ronparent
June 18th, 2010, 11:11 PM
Now we are getting somewhere. Is you system installed on the raid? If it is the best way may be to install the lvm2 and mdadm to the live cd session (like you mentioned in your prior posting) and then chroot (change root) to your / on the hard drive and install (apt-get) lvm2 and mdadm to that location.

If you are not booting for other reasons then we may have to explore finding boot parameters that would allow your system to boot. Post if you need more explicit help.

rsteinmetz70112
June 18th, 2010, 11:44 PM
I've been digging around in the /dev and have discovered something odd. First some background.

I have 4 hard drives and Disk Utility reports all are healthy. I can access them with fdisk among other utilities.

There are two raid arrays, and two SATA controllers with fake raid, but it is disabled in the bios. One controller is a SIL3512 (md1) the other is an nvidia.


/dev/md1 made up of all of sda and sdb
/dev/md2 made up of partitions sdc1 and sdd1

/dev/md1 contains volume group data which has 2 volumes (home and data)
/dev/md0 contains volume group remus which has 4 volumes, (root tmp, usr and var).

After recovering the config in with lvm, The volumes each have devices in /dev/mapper/

Also in /dev/mapper/ are three additional devices


nvidia_hfdafdba
nvidia_hfdafdba1
nvidia_hfdafdba2

These devices correspond to

/dev/sdc
/dev/sdc1
/dev/sdc2

There is no device /dev/sdc1, /dev/sdc2 /dev/sdd1 or /dev/sdd2, even though fdisk reports those partitions to exist.
/dev/sdc2 is /boot and /dev/sdd2 is swap,

The Disk Utility reports that /dev/md0 is degraded and I cannot access anything about it it with mdadm.

What are the nvidia devices and why is /dev/sdd no showing up?

ronparent
June 19th, 2010, 12:43 AM
The nvidia devices aappear to be part of a fakeraid array. But not necessarily so! For sure those drives were previously used by an nvidia raid chipset on a MB. the meta data written to those drives remains unless it is explicitly deleted. Since dmraid (the fakeraid linux 'driver') is installed automatically in 10.04 these drives will show up whether or not they are currently valid (and apparently whether or not the raid bios is active). If you are certain you never used a raid bios on your MB, then you can tempoarily hide them by booting the live cd with the parameter 'nodmraid' added to the live cd boot line (an f6 option). With dmraid sidelined you can hopefully work out the rest of your problems and get dmraid permanently uninstalled later. If you are not using fakeraid be careful you don't turn on a MB raid or you could destroy your partition tables. Hopefully everything will turn out fine but the outcome is somewhat unpredictable.

rsteinmetz70112
June 22nd, 2010, 05:18 PM
Thanks, thats very helpful. I've never been sure how all of these different things interacted.

I am out of the office this week at another site. Next week I'll get back to this.

I have another question;

Why does installing mdadm also install postfix?

To Review.

As I understand what will probably work is to;


Boot the Live CD <F6> nodmraid
install mdadm
Create md devices
install lvm2
recover Logical Volume Groups
mount remus-root on /mnt
mount the other filesystems in the appropriate directories
chroot to /mnt
uninstal lvm (if installed)
install lvm2 (and mdadm if not already installed)
attempt to boot.

It seems probable that the Grub command line will need to be modified to not use dmraid as well.

ronparent
June 23rd, 2010, 08:31 PM
It seems probable that the Grub command line will need to be modified to not use dmraid as well.

You are correct. You may also want to uninstall and purge dmraid as well as erase the 'fakerade' meta data ( ie 'sudo dmraid -r -E /dev/sdx' etc).

wheniwork
June 25th, 2010, 01:44 AM
Now we are getting somewhere. Is you system installed on the raid? If it is the best way may be to install the lvm2 and mdadm to the live cd session (like you mentioned in your prior posting) and then chroot (change root) to your / on the hard drive and install (apt-get) lvm2 and mdadm to that location.

If you are not booting for other reasons then we may have to explore finding boot parameters that would allow your system to boot. Post if you need more explicit help.

I'm having the same exact problem. Upgraded to 10.04 and now my os installed mirrored LVM raid drive won't mount.

Can you provide me with a bit more detail on how to install this stuff to my actual raid device while booted to LIVE cd?

Thanks!

wheniwork
June 25th, 2010, 02:33 AM
I was able to mount the LVM array to /media/system while booted to the live CD. All data is there. Now how do I get lvm2 loaded onto the raid device (/media/system) so it will boot?

You mentioned chrooting but I am not sure what or how to do that.

Update. Tried chrooting to the mounted system and installed LVM2. Got this output. Not sure it did it right


root@ubuntu:/# apt-get install lvm2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libx264-79 nvidia-180-libvdpau
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
lvm2
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/409kB of archives.
After this operation, 1,114kB of additional disk space will be used.
Can not write log, openpty() failed (/dev/pts not mounted?)
Selecting previously deselected package lvm2.
(Reading database ... 182155 files and directories currently installed.)
Unpacking lvm2 (from .../lvm2_2.02.54-1ubuntu4_i386.deb) ...
Processing triggers for man-db ...
Can not write log, openpty() failed (/dev/pts not mounted?)
Setting up lvm2 (2.02.54-1ubuntu4) ...
update-initramfs: deferring update (trigger activated)

Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-22-generic-pae
grep: /proc/modules: No such file or directory
grep: /proc/modules: No such file or directory
grep: /proc/modules: No such file or directory
ERROR lilo fails for new /boot/initrd.img-2.6.32-22-generic-pae:

Warning: LBA32 addressing assumed
Fatal: raid_setup: stat("/dev/md0")
dpkg: error processing initramfs-tools (--configure):
subprocess installed post-installation script returned error exit status 1
/bin/df: `/proc/sys/fs/binfmt_misc': No such file or directory
Errors were encountered while processing:
initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

wheniwork
June 26th, 2010, 07:09 AM
Sorry to be a bother, but if anyone can help out I would appreciate it. It sucks not having my server running and I'm at a bit of a loss as to how to get my RAID1 LVM device booting again.

I can assemble the raid and mount it just fine when booting to the live CD. Just don't know how to get it to boot after upgrading from 9.10 to 10.04.

My posts start on page 2.

Thanks! -Chad

wheniwork
June 26th, 2010, 10:24 PM
*bump*

Also, here is an image of what the screen looks like when it drops to shell.

Please note that my boot array is md0 and is RAID1

and md1 is my storage array and is RAID5

http://www.wheniwork.com/other/photo.jpg

rsteinmetz70112
June 28th, 2010, 06:23 PM
OK, I'm back in town and have been able to boot from the Desktop CD with "nodmraid". Still can't boot from the hard drive although the devices all seem to be there.

After booting I open a terminal;


$ sudo bash
# apt-get install lvm2
# apt-get install mdadm
# mdadm -A --scan
# lvm
lvm> vgcfgrestore remus
lvm> vgcfgrestore data
lvm> quit
# mount /dev/mapper/remus-root /mnt
# mount /dev/mapper/remus-usr /mnt/usr
# mount /dev/mapper/remus-tmp /mnt/tmp
# mount /dev/mapper/remus-var /mnt/var
# mount /dev/mapper/data-home /mnt/home
# mount /dev/mapper/data-files /mnt/files
# mount /dev/sdc2 /mnt/boot
# chroot /mnt
# apt-get install lvm2
lvm2 already installed



I'm obviously doing something wrong.

wheniwork
June 28th, 2010, 11:26 PM
Ok. I've made some progress. Turns out the mdadm.conf file was identifying the array per the drive letters. ie /dev/sdX

It should just have been the UUID.

So now the system mounts the LVM RAID1 device and starts to boot.

The problem now is that the UBUNTU 10.04 load screen just hangs and sits there. If I look at the terminal its sitting on the line.

"filesystem seems mounted read-only. Skipping journal replay."

Just sits there and nothing happpens. I let it sit on that for like 5 hours thinking it was doing something.

Any ideas?

rsteinmetz70112
June 29th, 2010, 03:42 PM
I checked my Mdadm.conf file and it seem to be correct. arrays are identified by UUID.

I've also just realized that Grub was no upgraded when I upgraded the operating system. That may have something to do with it.

wheniwork
June 29th, 2010, 03:52 PM
I deceided to simply format and reinstall. I had a backup. So setting up my settings will take about he same amount of time as figuring this thing out.

rsteinmetz70112
June 29th, 2010, 05:59 PM
I have several similar servers to upgrade so I'm going to keep at it. Beside I'll eventually learn something.

I hope someone come along with some additioanl information soon.

vpablos
September 1st, 2010, 09:57 AM
We are having similar problems since last friday.

Array was degraded so we told the system to include those partitions that were not detected as part of the array (raid-1). After it was rebuilt we reboot the machine and now it is not bootable.

We've tried with installation disk but it does not create special decives md0 and md1. We've tried installing mdadm and lvm2 and modprobing dm-raid as suggested in some links.


Now I have found some other usefull links like
http://ubuntuforums.org/showthread.php?t=990045
http://wwww.ubuntuforums.org/showthread.php?p=4915572

And here they say you need Ubuntu server CD ... I'll give it a try
http://blog.edsantiago.com/articles/ubuntu-raid-luks-lvm.html

but, do you know if we can use knoppix live cd to recover the disk contents or it is too much dangerous to use a different os?
http://www.howtoforge.com/recover_data_from_raid_lvm_partitions

This one is the most complete I've found:
http://pve.proxmox.com/wiki/SMEServer_LVM_Recovery_using_Knoppix_LiveCD