Help need to fix server disk errors.

[TheFu]

TheFu,

I think you're thinking of "LHammonds".

Yep! He's still around here too. That's good news! I've had too many online friends disappear this year, and a few IRL friends too. ;(

https://ubuntuforums.org/showthread....5#post14028065 has a link, but it won't open for me. Maybe the wayback machine has it or it is down temporarily? IDK. This sort of stuff is why I run a Wallabag Server - so when things I want as reference are found on the internet, I can grab a local copy that can't disappear. I didn't snag anything from LHammond as our skills overlap, sorry. I did snag a bunch of LVM articles, but mostly around snapshots, thin provisioning and use with LXD. None of that is helpful for this thread.

[sgt-mike]

ZFS vs MDADM raid might be a option that the OP might consider, when /if rebuilding.
Not sure what the OP's objectives are. (I mean we know he wants to fix the array, but rather long term storage redundancy needs 1, 2, or 3 disk failure)

I have ran both I did like mdadm in a raid 5 array, but ZFS seems to be way easier to setup / maintain, but then again I haven't ran it for years yet. Like some on here have. But so far it's been great. Both have strong points.
I went 9 drives wide @ 4TB a drive with a ZFS pool and will soon add at least three hot spares to that data pool.

My post doesn't fix the OP issues but does provide food for thought for the OP in a planning stage.
And yes you can add hotspares to a mdadm raid 1 array. Which is the take away. Hotspares allows time to get a new drive into the array/pool.

Like the The Fu says raid is not a backup. Personally I store backups and snapshots separately. I'm constantly (as in daily) going to my NFS server via ssh to check the drives health. Only thing a raid does is allows you time to get a drive in before 1 drive failure turns into two or three failing drives, other than get a group of drives to act as one. Primary idea is to replace the drive before it actually fails.

The Fu will know way more than I on how to fix this to work with LVM. I Just wanted to throw the hotspare idea out there.

[rebeccatucker]

Great

[tross9]

update: ( I am going to replace the sdb drive but I want to test a few things first)

1) it looks like the Mysql database on sdb is sitting on a badblock, when I stop the mysql pgm the server stop throwing sector errors.
when I stop mysql, the /dev/sdb no longer appears in the df -h but /dev/md127 is still listed. (md127 is the raid of sdb)

what i would like to do.

1) fix the bad sectors/blocks so the server stops throwing errors.

optional - remove the raid ( drop md127 leaving sdb without formatting, if possible, this can wait till the new drive is in)

so far I did the following , but not sure if it fixed the errors.

1) /etc/init.d/mysql stop --- this stop the server from throwing errors, if i did not stop mysql , after about 10 - 20 minutes sdb not md127 would not display in the output of df -h
2) sudo badblocks -sv /dev/md127 > badblocksoutput -- listed error (20/0/0)
3) sudo fsck -t ext4 -L badblocksoutput /dev/md127 ---- ran to completion
4) rebooted server
5) still throws sector errors. ( until mysql is stopped)

If I understand FSCK correctly with the option used above, it should have tried to mark/add the bad blocks to a list and try to move the data if possible, i think it may have marked the blocks but mysql is still pointing to the bad blocks (step 5) or fsck did not mark the blocks.

not sure what to do now, assuming the blocks are mark, how can I check to see if the server has mark the bad blocks or how to manually mark them , if did not mark them.


TIA

Tim

fyi -- at the time the Raid was setup as a temp backup solution till I could get a permanent solution in place and not knowing how to remove the raid without loosing data I left it as is.

"Only thing a raid does is allows you time to get a drive in before 1 drive failure turns into two or three failing drives," -- that was my original idea for the raid.

[TheFu]

Hard disks have extra sectors on them that are automatically remapped so the disk will continue to work. If those spares run out, time for a replacement. Period.

Use SMART data to see how many and how quickly the reallocations are happening.

[tross9]

I have to apologize, I failed to state that when mysql is trying to load, it is throwing only one sector error , it displays the same sector over and over until I stop mysql.

what I'm look for is
1) what disk utilities I need to run to check and fix drive errors, which option to use, and which order to run the utilities, if more than one is needed.
2) how to tell if the bad blocks and sectors have been marked so that the system knows not to use them.

this Ubuntu server is a Home system and I'm using it to try to learn how to setup one up correctly and maintain it correctly, and not doing it that well . (note: man pages are not my friend, utube vids work better for me)
the long term goal ( when I have the money for this) is to have it as a database, file shares and cloud server for my windows Pc's. and back up the server to an external device. for now just a DB and testing server.

until you mentioned smart monitor, I had no clue it existed, and I'm not sure how to use it correctly.

again thanks for all of the replies

Tim

[TheFu]

Google found this: https://forums.linuxmint.com/viewtopic.php?t=255457
and
https://www.tecmint.com/check-linux-...rs-bad-blocks/ I've found Tecmint is a reputable site for Linux stuff.

If you want a DBMS backup, you'll need to use some smart methods to get a clean backup that isn't corrupted, since DBMS files are always open and being modified. Stopping the DB is 1 method. There are a few others that don't require stopping the DBMS.

BTW, MySQL has been out of favor for nearly a decade now. MariaDB replace it. You can learn about the history of those changes, if you care. They are API compatible and the team behind MariaDB were the original creators who chose to leave Oracle.

[tross9]

thank you, I'll look at these.