Converting desktop into server

[TheFu]

There are times when daily monitoring of logs isn't sufficient.

I've had log files explode to use 15G in less than 1 hour.

[ameinild]

I've had log files explode to use 15G in less than 1 hour.

That's pretty insane !

[mastablasta]

That's pretty insane !

and that's nothing. error in Steam caused error log to fill up the home folder that is 950 GB in size (ok about 600 GB is still free) in 5 hours. so that's over 100 GB per hour. and no one noticed until it home was full as we didn't touch the PC. busy machine, interesting bug.

EDIT: since i had separate home, the solution was simple as boot was still possible (root was not affected). all i had to do was locate and remove the offending file.

on a personal or hobby server this is not much of an issue indeed. however it is easier to recover if you have /home and /var/log on separate partition.. but still on home server you can always boot live, delete the files and reboot or reinstall. any down time due to errors probably doesn't matter so much there.

[ameinild]

Specifically log files under the /var can go wild if you enable some services to debug in verbose mode and forget to turn it off...

I've had log files explode to use 15G in less than 1 hour.

Related question: Wouldn't it be possibly to also control the size of logs with filesystem quotas, as described in this article?

https://www.digitalocean.com/communi...n-ubuntu-18-04

The folder /var/log is owned by the group "syslog". If you set a quota for this group with a hard limit of 2 GB or something, wouldn't this have a similar effect?

I know this is only for logs, but maybe it could be used in a less critical setup, where you want to limit the sudden growth of logfiles, and still would prefer a simpler partition setup.

Combined with the root reserved space in tune2fs, I believe this would be a rather good option - haven't tested it though.

[LHammonds]

Related question: Wouldn't it be possibly to also control the size of logs with filesystem quotas

Possible? Yes.

The folder /var/log is owned by the group "syslog". If you set a quota for this group with a hard limit of 2 GB or something, wouldn't this have a similar effect?

Good question...I don't know for 100% certainty. There are files in /var/log that belong to root:root, root:adm, root:utmp, syslog:adm, landscape:landscape, root:systemd-journel. I would imagine quotas only apply to files associated to that user or group. I doubt it considers the owner of root:root for "/" and assumes everything on the disk is assigned to root:root just because it is the top-level user/group.

Quotas are defined in 3 ways: Users, Groups or FileSystems.

If you have a single, atomic partition scheme and you place a limit on a User, that limit applies to everything on that system, it cannot be confined to just /home or whatnot. Same for Groups. FileSystems aspect seems completely useless unless you have multiple partitions which would then allow you to have different limitations based on partition. But if you are using separate partitions, you are splitting hairs at this point and possibly introducing complexity and overhead that is not needed.

You also need to run quota checks to make sure it does not get "outta whack" such as after an unexpected reboot, etc.

Even if you are mainly just combating logs, you cannot expect to see all logs in /var/log. Applications can put logs elsewhere such as in /opt or /home or /var/lib (such as MySQL/MariaDB binary or general logs). You might get all the log-using scenarios today but a year from now, if you install a new application and forget about the quota setup, you can experience a "gotcha" moment that bypasses all your hard work with a log file outside your quota setup.

To me, using quotas add more complexity than isolating the root partition from others via partitions...partitioning requires less overhead in performance, maintenance and gotcha cases.

There certainly are scenarios where quotas are extremely helpful such as multi-user systems (ISP FTP/Web/DB, File Server, etc.), but for general purpose, I prefer the set-n-forget method of partitioning. Every server I deploy uses multiple partitions. If I ever need to utilize quotas for a specific scenario, I still would not revert to a single, atomic partition.

LHammonds

[ameinild]

Thanks for the thoughts!

[TheFu]

Log file size can be managed by logrotate. This is built-into all Ubuntu releases and many other distros. Check out /etc/logrotate for example configuration files. Additionally, journalctl has a method to limit log file sizes. /etc/systemd/journald.conf is the config file for that.

I wouldn't use quotas for OS limitations, only for end users, who actually would take action to clean up their cruft. Quotas on the OS would just hide a big issue from the admin, who needs to address the problem. When /var/log fills up and refuses to accept any more data, I think the system would crash. It certainly will become very slow.

[ameinild]

Additionally, journalctl has a method to limit log file sizes. /etc/systemd/journald.conf is the config file for that.

So here you're actually saying that journalctl is much more well thought through than "standard" Linux journalling, since you can actually limit the amount of log spam in an intelligent way, whereas "default" Linux config can crash an entire system by log spamming, if not devising a clever partitioning scheme, or maybe quotas? This is interesting.

When /var/log fills up and refuses to accept any more data, I think the system would crash. It certainly will become very slow.

But isn't this the same when the log partition is only 2 GB and suddenly fills up? I'm trying to compare the effects of the 2 different scenarios to manage log file size.

I actually find it quite baffling that apparently the default log system in Linux is so fragile that it can't easily be configured to limit the size of logs, and that you have to go through hops and loops to devise a method to limit a system crash. IMO it would be better to design the system so you would never be in a case where the logs fill your entire disk, which I think everybody can agree is a bad situation, that could rather easily be prevented (for instance by never allowing logs to consume more than a fixed amount of the free space).

[TheFu]

logrotate has worked for 20-40 yrs, limiting the size of log files. Misconfiguration is the most common issue.

There is no a central "Linux team" somewhere deciding things. Different Linux distros have different methods of controlling log file growth and limits. I prefer the logrotate method, but that is mainly because I've used it for decades and it works for logs that are actually managed through it.

If a 3rd party program doesn't tell logrotate about the log files it uses, I can't blame it for not managing size limits. The same would apply to journalctl. If it doesn't know about a log file, it can't monitor and limit file sizes. Even when I've setup journalctl to limit file sizes, it hasn't followed that limit. I assume the issue is user/config error, because I wanted logs rotated every week regardless of the size and only rotated based on the size when they hit the limit specified. I like to keep 26 weeks of all logs. Call me crazy.

Journalctl does move some external stuff commonly used into the log system. Eventually, that will be good. Now it still feels like an extra middleman in our logging processes.

/var/ isn't just a log partition. There are many other things there. As usual, something that seems simple, isn't.
If logs are going to fill a system and nobody is looking at those logs, I would argue that crashing the box to get attention is a desired behavior. Others can disagree.

[ameinild]

Thanks for insights as always. But as you mention yourself, I guess logrotate can't properly handle a sudden log increase to 15 GB / hour before filling the log partition on most systems - or maybe it can?
Anyway, all I'm saying is that, as a contrast to making separate partitions for /var/log, there could be another space controlling mechanism to ensure that abnormal log generation on it's own wouldn't crash a Linux server.
It sounds like server admins are generally relying on partitioning to solve this issue. On the other hand it sounds like a rather "serious issue", where I don't understand why there isn't a more straightforward and easier solution, that doesn't impose the other limitations that multiple partitions do.

Also, the reason I'm focused on logs is 1) because you and LHammonds bought it up as an explanation for multiple partitions and 2) because almost every post on Askubuntu where a user's system suddenly has no space left is because some insane logging spamming has been in effect. So it's apparently also an issue for many "normal" users that just want to use their PC.

After digging a little more into rsyslogd, it also seems you can limit the size of particular logs. Maybe it would be appropriate that Ubuntu had a default max syslog size of 2GB.

It's ok that we disagree. I maybe just think that Linux (and Ubuntu in particular) could be a little easier for everyone if there was a more straightforward way (and maybe set up as default) to prevent that your system was brought down because of log spamming. And since /var/log is the default location for this, I think that limiting the total size of this dir or the standard log files is a good place to start - even if you have the files on a separate partition.