Partition table messed up - hundreds of "ghost" partitions

[YesWeCan]

I am not knowledgeable enough to discern much from the fsck for sda8. Inode 7 is toast and there are free block count mismatches. Obviously, don't try to fix it while it is mounted.

You could make a new 40GB partition on the new drive and copy the contents of sda8 to it before you repair it.

sudo rsync -vax /dev/sda8 /dev/sdbX

I'd probably just repair it.

When all this is working well enough, for absolute certainty, I would suggest you boot a live CD/USB and reformat the new drive, then manually create partitions on it into which you can copy the contents of the old partitions. This way you end up with perfect file systems. Then adjust the UUIDs in fstab files and reinstall Grub. Then you can dd clone the new drive back to the 500GB one. This will take a lot of time and I don't know whether this is strictly necessary.

I recommend rsync -ax --progress for copying partition contents

[Quaxo76]

Lesson learned? This is why I recommended manually finding the EBRs and fixing them.

I know... I was getting anxious and impatient and it looked like testdisk knew what it was doing, and its partition table seemed to make sense... Who knew it would change start and end point like that! :/

Cristian

[YesWeCan]

I know... I was getting anxious and impatient and it looked like testdisk knew what it was doing, and its partition table seemed to make sense... Who knew it would change start and end point like that! :/

Cristian

You weren't to know. You have to be very careful about using automated tools on critical data. And you have to carefully vet recommendations you receive to make sure the tools are proven to do the job properly. TestDisk is an experts tool and is not smart enough to know which partition remnants are the current ones. But what really worries me is that it created an illegal extended partition end sector! There is no excuse for that. So treat all tools with healthy cynicism - don't assume the people who wrote them were competent.

And do regular backups from now on
[edit] Oops, I forgot you were doing that before! Sorry. Ok, from now on do not let anyone steal your backup drive.

[Quaxo76]

Ok, from now on do not let anyone steal your backup drive.

Hehe... I was keeping it in my locker at work, because I thought that would be safer than carrying it around with me every day, with the risk of dropping or loosing it... Maybe I should keep *two* backups, one at home and one always with me!

Repartitioning the drive makes sense because even before having this problem, GParted always warned me that the partitions' boundaries were not aligned with cylinders, possibly ending in poor performance. Every attempt at re-aligning without totally repartitioning failed... (they had become misaligned when I cloned my old, smaller, HD into the 500GB one). But: does RSync preserve ownership, permissions, etc. on files? After moving the files from the old, rescued, partition to the newly created one, does it just work after reinstalling Grub?

Now, to go back to the start... This all started when I tried to revive an old Windows partition (I need to use Windows for a few days, to interface with a certain hardware). If I decide to just wipe sda1 (the old Windows) and reinstall a new WinXP onto it... it will of course destroy GRUB, but is it safe to do so and then reinstall Grub? Or will I find myself deep in trouble again?

Cristian

[YesWeCan]

Hehe... I was keeping it in my locker at work, because I thought that would be safer than carrying it around with me every day, with the risk of dropping or loosing it... Maybe I should keep *two* backups, one at home and one always with me!

I back mine up to a USB drive but I encrypted the whole drive in case I lose it: http://www.emcken.dk/weblog/archives...in-ubuntu.html
"The Cloud" sounds like a good alternative but you'd need fast internet connection and you'd probably have to pay for the capacity you need. I haven't used it and I don't know how secure it is...I would encrypt my files first.

Repartitioning the drive makes sense because even before having this problem, GParted always warned me that the partitions' boundaries were not aligned with cylinders, possibly ending in poor performance. Every attempt at re-aligning without totally repartitioning failed... (they had become misaligned when I cloned my old, smaller, HD into the 500GB one).

Actually, cylinders do not exist any more as such. That is to say the boundaries are artificial now. An important thing is to align partition boundaries with physical sectors rather than logical sectors if they are different. In the past, drives had both as 512 bytes but newer ones are now using bigger physical sectors, like 4096, to improve performance. The logical sectors are still 512 so the OS is happy. fdisk will tell you what your drive has.

But: does RSync preserve ownership, permissions, etc. on files? After moving the files from the old, rescued, partition to the newly created one, does it just work after reinstalling Grub?

Yes rsync -a preserves everything and you can stop it mid-copy and then when you resume it, it carries on where it left off. Very useful tool.
After copying, the UUIDs of the new partitions will be different and this means the fstab(s) has to be adjusted and Grub has to be reinstalled.

Now, to go back to the start... This all started when I tried to revive an old Windows partition (I need to use Windows for a few days, to interface with a certain hardware). If I decide to just wipe sda1 (the old Windows) and reinstall a new WinXP onto it... it will of course destroy GRUB, but is it safe to do so and then reinstall Grub? Or will I find myself deep in trouble again?

It should be ok to do that.
The other thing you could do is install Grub to another drive, like a USB stick, first. Then you could boot the USB to get to your OSs while leaving the standard MBR which XP will install in sda.

Remember, don't have the new drive connected when you boot because currently it has duplicate UUIDs and this may cause the wrong partitions to be mounted.

[Quaxo76]

So... I backed up sda8 with dd, then ran fsck on it. And it reports clean now, and the files still seem accessible. I can't say for sure if *all* the files are there, but things are looking good.
Then I fixed the old system's fstab (to reflect the recent changes) and tried booting it, as all the filesystems were reporting clean... And it works! I'm writing from my (revived) old partition now. Now I'll repartition the new drive and move the files there. I might use the new larger drive in the laptop and use the older as a backup...
I'm marking the thread as SOLVED now... and I want to thank again everyone who helped me fix my problem! I got my files back and learned a lot in the process.

I back mine up to a USB drive but I encrypted the whole drive in case I lose it: http://www.emcken.dk/weblog/archives...in-ubuntu.html
"The Cloud" sounds like a good alternative but you'd need fast internet connection and you'd probably have to pay for the capacity you need.

Actually, I'm not overly concerned about security - I never store passwords or sensitive data on the data disks. It was mostly technical books that I had gathered in the years, or programs I had written, or other sorts of personal data. So, losing it is bad for me because I wouldn't have it anymore, not because anyone else would be interested in it... and btw I'm 99% sure that who stole my old hd just wiped it and resold it (it wasn't even FAT or NTFS, so it wouldn't show up in Windows, and I don't think the average thief knows about ext4!)

An important thing is to align partition boundaries with physical sectors rather than logical sectors if they are different.

Yes, that's what it said, sorry, my mistake. The drive's physical sectors are 4KB while the logical sectors are 512bytes.

The other thing you could do is install Grub to another drive, like a USB stick, first. Then you could boot the USB to get to your OSs while leaving the standard MBR which XP will install in sda.

I think I can also just backup and restore the MBR, right? If I prepare the partition for Windows first, with GParted, then do

dd if=/dev/sda of=/mbr.bin bs=446 count=1

then install Win, then boot a livecd, mount root, and restore the mbr, then re-setup grub so it "sees" the new Windows... it should work, right?

Thanks again
Cristian

[YesWeCan]

I think I can also just backup and restore the MBR, right? If I prepare the partition for Windows first, with GParted, then do

dd if=/dev/sda of=/mbr.bin bs=446 count=1

then install Win, then boot a livecd, mount root, and restore the mbr, then re-setup grub so it "sees" the new Windows... it should work, right?

Thanks again
Cristian

That should work. I normally advise people to do a live CD re-install to avoid the perils of a accident in the dd command.
sudo mount /dev/sda2 /mnt
sudo grub-install --root-directory=/mnt /dev/sda
in your case.

Glad you sorted it out.

[srs5694]

But what really worries me is that it created an illegal extended partition end sector! There is no excuse for that.

This is a known problem with TestDisk. Some background (for Quaxo76; I expect you know this, YesWeCan): ~20 years ago, hard disks had a fixed number of cylinders, heads, and sectors, so disks' addressable space would be an exact multiple of a whole number of cylinders, heads, and sectors. Years ago, though, disk geometries became more complex, with a variable number of sectors per cylinder, so disks' firmware began producing CHS values that were convenient fictions. One consequence of this is that there were usually some "leftover" sectors after the last whole cylinder. Using CHS addressing, these were unusable, but using the newer LBA addressing, they're usable, and modern partitioning tools often do use them.

Based on tests I've performed, I suspect that TestDisk uses CHS addressing internally, and when it finds a filesystem that ends in this "leftover" area between the end of the last whole cylinder and the actual end of the disk, it creates an extended partition to hold the logical partition that's the size of a whole cylinder -- that is, too big for the disk. This is obviously a bug.

Fortunately, most OS kernels can handle such disks OK, and there's no risk of data loss in normal filesystem operations, since the logical partition that TestDisk creates is properly sized. Such a layout is technically illegal, though, and GParted tends to react to illegal disk layouts by displaying an empty partition table.

This problem can be corrected by using sfdisk to resize the extended partition, as YesWeCan directed you to do; or by using my FixParts program, which rebuilds a fresh extended partition without using CHS values. In theory you could fix it with fdisk, but it would be tedious, since you'd need to re-create all your logical partitions. There are probably other non-Linux programs that can fix the problem, but I don't know what they are.

Of course, this is just a tiny part of the problem you encountered. I don't know what caused the infinite loop of EBR links that set off the whole thing. FWIW and IMO, the linked-list data structure of logical partitions is a weakness because such problems can occur with it. Most other partition table types, such as the Apple Partition Map (APM) or the GUID Partition Table (GPT), could not suffer from that specific problem, since they don't use linked-list data structures.

[srs5694]

One other thing: In addition to regular data backups, keeping a backup of your partition table will enable easier recover if a problem like this recurs. The type of sfdisk output you've used to edit partition start and end points will do the trick:

sudo sfdisk -d /dev/sda > parts-sda.txt

If the partition table gets messed up, you can restore this to fix the problem. Be sure to keep your backup up to date, though; if you resize partitions with GParted, change partitions to install another OS, or whatever, restoring the old partition table will trash your changes!