Ubuntu 16.04 Sever freezes every day

[manuel42]

Hi,
I'm having lots of trouble with a server running UBUNTU Server 16.04. It freezes in a daily basis, mostly at night, when there are not users connected or processes running.

I don't have any clue about what could be the cause. I've already run memtest (no errors detected) and can't find nothing relevant on logs (I guess someone with appropiate skills could find something).

When my headless server freezes, it can behave in different ways:

- No SSH access at all (it asks for user/pass but then doesn't respond)
- Samba is out. SSH works very slowly. Power off or reboot commands doesn't respond. Systemd process is consuming huge amount of CPU but Kill the process doesn't work. Sometimes (very rare) even file system turns read-only ¿¿??

Any advice about where could I look for some clues about what is happening would be appreciated.

Thanks!

[mastablasta]

Sometimes (very rare) even file system turns read-only ¿¿??

it goes into read only mode if there is an issue on disk. run smart to check the system disk state (disk might be failing).

[manuel42]

Hi mastablasta,

Thanks for the hint. This server has two drives so I've checked both using smartctl (long and short tests): "No Errors Logged".



Regards.

[Graham_Willis]

Just copy the system to a VM, let it run for some time and check if you'll observe the same behaviour. If not - try to generate the same kind of traffic that's going on on the production machine and re-check the results. If the copied system behaves strangely, then you already know that it's a software problem. If not - then it might be or it might be not. But then, if it's feasible. try to cut off all the traffic on the firewall at night and see if it helps. If doesn't - try to killing processes (before the problem appears), perhaps you have a rogue service.

[gordintoronto]

What motherboard? I have a 7-year-old Gigabyte board which requires acpi=off.

Do you monitor CPU and memory usage?

[MAFoElffen]

Freezing on what hardware and running what services? (make, Model, CPU, RAM, Disk spec's; Services)

From the previous posts, I gather that you are running at elast Samba(?) or at least connected to a Samba Share.

On servers that freeze intermittently, or no apparent reason-- CPU, RAM, Disk. Hard to diagnose a frozen machine when it is frozen, so check or generate logs to determine the events leading up to the frozen state. Of course, you need to check that you do not have a full disk, before you can generate additional logging, or else you just add to that problem.

Does that make sense? Those are hints, which can be expanded upon, if needed.

[manuel42]

Thanks for your help Graham_Willis, gordintoronto and MAFoElffen.

This is a 1 year old ASUS desktop machine, 8 GB ram, 2TB HDD (with at least 1.8 free). This issues began after upgrade to 16.04. There where no issues with 14.04.

RAM tests where OK, as well as HDD tests. CPU and RAM usage are never over 50%.

This server is used as fileserver (Samba) and as a MySQL server. Today I'm going to try stopping Samba and check if it still freezes.

I guess I'll downgrade this server this week if I don't find how to fix it.

Thanks again. Your help is sincerely appreciated.

Regards.

[mastablasta]

Thanks for the hint. This server has two drives so I've checked both using smartctl (long and short tests): "No Errors Logged".

.

you can use various diagnostic distros. i use ultimatebootcd to do various tests.

SMART status might be ok but you should check the values. if they are too much off for a young hard disk then it could indicate issues eventhough smart check is good. so far i had only 2 issues - small sample, so i am maybe not qualified to talk too much about this. one disk was failing with cliking and i couldn't boot always, smart was mostly good, one number was a bit off. exchanging the disk solved the issue. next one was flash driver which keep dropping into read only. checking it with smart tools didn't reveal anything. specialised USB check software showed errors. so i had to resque the os from it and copy what i managed to save to new USB flash drive. all works well now.

[MAFoElffen]

Since you have verified that you have good disk, and plenty of disk... then create a cron job to append the top-5 stats from TOP, to a log file, every 5 minutes. Once a day, "move" (rename the file with the date) the file for review and cleanup.

That way you can see what the system was doing before it crashes, leading up to that crash. That way you can see if an application was building up on resources that it was not releasing... Like I said, you cannot get those machine sate stats from "memory" from a frozen machine, but if you log them, so they stay as persistent... another would be to check journalctl.

Those are things I do, when trying to track something like that.. Of course, some crashes are just intermittently rouge.(and frustrating when trying to track down.) You could always use a management utility to track and log. But you would essentially be doing the same right?

[manuel42]

I'm learning a lot these days Thanks again. I know I'll find out the issue and fix it.

I've just found a new BIOs update to fix stability issues, so I've updated it. I'm also doing some research about diagnostic distros as mastablasta suggested, as well as the cron job to record stats until system halts (good one MAFoElffen!).

Maybe it's just fixed with the bios update. We'll know tomorrow!

Regards.