*rant warning* HELP: How do I come to terms with systemd?

[egorFiNE]

Help me get over it.

I hate it. I hate everything about it.

I'm far from being a luddite either: I love new things. Upgrading dependencies in my company is done daily. I run bleeing edge hardware and software since nineties and ubuntu since version 6.

But all I can see in systemd is cancer. It seems to me they are on a constant lookout for the most stable things that worked perfectly well for decades - and they break it.

I do not understand `ps ax` anymore.

Having said that, I have to admit that there are many serious arguments in favor of systemd infrastructure. Some of the classic systemd complaints are no longer true and some of reasons why systemd things were invented are indeed valid. systemd as PID1 was a good idea. I like it. I won't go back to sysv scripts.

Problem is, even PID1 is swelling. We now have timers, mounts, socket activations and other things I couldn't even think of would become a part of init system. Up next: systemd-ls. Don't cringe yet: nobody could have imagined systemd going after `sudo` or tmp files garbage collector either.

Now, I perfectly realize I cannot be smarter than the absolute majority of Linux distro devs. They choose systemd-* time and again despite the fact that it goes against everything Linux was loved for. systemd-$SOMETHING is notoriously known to be so much broken yet it is included in the default Ubuntu install release after release.

So my question is: how do I get over it and come to terms with the world of ever growing systemd?

I'm sure I'm not the only one who shared this view and changed and came to love (tolerate) systemd. Please help me change.

PS: No, running a systemd-free distro is not an option anymore in 2024.
PPS: Is this is the right approach?

[Tadaen_Sylvermane]

While there are many perspectives on the whole systemd thing, and forward movement in general none are particularly right or wrong. Let's look at it from a practical point of view.

--

Does systemd in this case have any real effect on your operation of your computer at home? (this is mainly home users around here so I target them). Does it even matter when 99.99% of users don't even have any reason to interact directly with it?

Yes things are bloating a bit, but hardware is growing exponentially faster. At what point is worrying about "bloat" a concern worth any thought in normal day to day operation?

Growing pains are normal. People are generally resistant to change. While the majority of us are armchair enthusiasts with Linux, and in some cases at the professional level do any of us really fully understand this stuff? Linux has been taken over by big business, we just benefit from it. Linux users outside of the professional world are little more than a rounding error. As such do we have any real right to demand or fight against something that has probably had countless studies and changes done to it by these same companies who have infinitely more of their livelihood riding on this software?

If systemd in this case is such a big problem there is nothing stopping people from forking everything and going down another path with it. Divergence happens. The problem is that there simply aren't enough people interested in removing systemd for this to work. Is it really okay for that small minority to demand that the companies who primarily financially back the software we all use daily add this extra work for themselves with little to no real benefit to them?

--

I don't have a personal preference. I just use what I use. I don't interact with systemd myself outside of an initial installation where I put my backup service and timer in place. So I don't have a dog in the fight or argument. But these things are philosophical and can get emotional for some reason. Take the emotion out of the decision. Look at it strictly from a practical point of view. I neither hate it nor like it. It makes my computer work, that is all I need to know.

[#&thj^%]

Help me get over it.

I hate it. I hate everything about it.

Some other suggestions then: https://itsfoss.com/systemd-free-distros/

I remember feeling the very same way you now feel, but i chose to go along with it. (Now it's not so bad for my use)

Tadaen_Sylvermane has a fresh persepctive above.

[HermanAB]

Simple, install Devuan and go on with your life.

[ajgreeny]

Thread moved to Ubuntu, Linux and OS Chat. which is more appropriate for the discussion and should be a better fit.

[grahammechanical]

You have identified the reason a lot of Debian developers did not want systemd to be part of Debian. There was a big argument about it. Redhat is the developer of systemd and Redhat developers working on Debian pushed for its adoption. Ubuntu developers working on Debian were ordered to stay out of it. The Redhat developers won the argument and the Debian developers who lost forked the code and produced Devuan.

Ubuntu inherited systemd from Debian.

Regards

[TheFu]

For large installations, systemd solves (or more accurately, tries to solve) issues related to boot time by doing things in parallel that weren't done in parallel before. For small installation with just a few HDDs connected and just a few NICs, this was never really an issue, but in some corporate environments, I've seen hundreds of disks mounted over a SAN to a single server. Booting those systems tool forever.

Additionally, the udev development team was burned out, so they found some people (the systemd folks) to take over their project. Since nobody else stepped up to do the work to keep udev going separately, it was only reasonable for the systemd guys to put their particular "stamp" of design ideas into udev.

Think of the init and udev systems like a 200 yr old house that had been redone every 50 yrs or so. You never know what's behind the walls until you open them up. Sometimes when looking behind the walls, you find treasures and sometimes you find a termite colony that has been ignored, but needs to be removed. As with all pest control companies, they each have different ideas for how to solve issues and as systemd started interfacing with more and more of the init things inside Linux, I'm positive they found some nasty code and other beautiful stuff.

Sometimes when you find a post behind a wall, it isn't easy to tell if it is just a spacer/decorative or if it is what holds that section of the house up. Same with Linux code. The people who created all that old code weren't dumb, but perhaps they didn't clearly document things in a way that younger people would understand? The loss of "institutional knowledge" happens everywhere. It happened with init and with X/Windows. Sometimes rather than try to modify code that you don't fully understand, it is better to start over and bring consistency to a new, active, project for code styles, documentation, and interfaces.

All that sound reasonable?

Now for my real feelings. I think systemd is an abomination that must be stopped. I disagree with most of the added interfaces and bloat. I've deathly fearful that we'll never really get away from systemd in my lifetime as it takes on more and more whims for what could be possible. I'm actually more afraid of Canonical's Snap packages, but that's a different issue.

OTOH, journald is a work of art. That is part of the systemd project and it makes so many things related to logging so much better. About a year ago, I came across a distro that wasn't creating text-based log files in /var/log/ any more. At first, it was a little shocking, then I used journalctl to get the information I needed and found it was all there, stored more efficiently, easier to find just the stuff I wanted, and very easy to setup a remote log server.

I was happy with init. I understood how to make the start/stop/status/restart scripts and how to handle dependencies. It all "just worked" for me and for my needs. I miss it. I find the systemd Unit files obtuse and hard to get to do what I intend. Seems there are too many options for dependencies built in. Perhaps someone with a better understanding for specifying the dependencies in Unit files would make it do exactly what they desire. I've never had that luck.

I'd add netplan yaml files to the list of "what the heck were they thinking" list. It took them 2 yrs after it became the default for non-trivial network setups to be possible AND actually work. Because it is tied into the cloud-init too, which is another abomination, it is now a core requirement for any Ubuntu Server. A little over a year ago I started moving from Ubuntu to other distros in a concerted way. Started with a desktop and it has been fine. My new servers and upgrades as Ubuntu system support ends will be migrating to alternate, but stable, distributions to avoid snaps and netplan. I'd like to avoid systemd-resolved and systemd-timed as well. Today, I manually purge those last 2 and install tools that actually work.

Perhaps in 10 more years, systemd will be stable enough for use in production. It took about 12 yrs for PulseAudio to become stable ... and now it is being replaced. Hopefully, we don't have to wait so long for systemd to be yanked from distros and for the Gnome project to stop dumbing down and bloating up their project.

I don't know if you feel better, but I do. Thanks!

[#&thj^%]

I don't know if you feel better, but I do. Thanks!

I needed this smile today, I love a well said rant.

[egorFiNE]

Thank you for the insightful reply!

At what point is worrying about "bloat" a concern worth any thought in normal day to day operation?

When I have a server or an instance and I have no idea what it is busy with. When I type ps ax and I cannot comprehend what's up. When I pull my hair out for an hour trying to move sshd to other port only to discover that they have changed sshd to socket activation and did that sneakily in a multitude of ways.

do any of us really fully understand this stuff?

Good point. One of the things that hurts is that I used to understand like everything in Linux. Before that - everything in MS-DOS. Everything in programming. But as systems grow and complexity grows a single cannot even be expected to understand everything in any single package except perhaps gzip. So following this line of thinking I guess maybe it's time to give up on the notion that an OS has to be transparent to me and `ps ax` along with networking configuration and cronjobs has to make sense to me. What do you think?

companies who have infinitely more of their livelihood riding on this software?

Well I am a company having livelihood on Linux. Not until recently I have managed a few thousands of physical servers. This is why I want insight and transparency because a mere 1% increase in performance means a hundred more servers is now available to take on more customers.

The problem is that there simply aren't enough people interested in removing systemd for this to work.

And this is where we circle back to my original point: I cannot comprehend why the majority of server folks are okay with systemd cancer. Practically. Why? Netplan is worse than ifupdown and systemd-resolved has been designed like a plastic wrap left on the dashboard of a new car meant to be removed by the owner.

But these things are philosophical and can get emotional for some reason. Take the emotion out of the decision.

True. It's just hard for me when another metastasis of systemd breaks things that both were working perfectly well and working for decades.

Look at it strictly from a practical point of view. I neither hate it nor like it.

And that's my main take away from this conversation. Simple words, they did help me. Thank you.

[egorFiNE]

For large installations, systemd solves (or more accurately, tries to solve) issues related to boot time by doing things in parallel

I full agree that as PID1 systemd is way better than the init scripts. I won't go back to sysv init anymore. Problem is: once developers created a perfect init, they were bored and started to look around and that's precisely where bad things started to happen.

I've seen hundreds of disks mounted over a SAN to a single server. Booting those systems tool forever.

I had something similar in production. The trick is extremely simple: you don't bring that up with an init system. You just don't. Let the init system up the system, up the networking, start ssh and let go. Then you are free to start up your business layer in whatever ways required.

Sometimes when you find a post behind a wall, it isn't easy to tell if it is just a spacer/decorative or if it is what holds that section of the house up.
[]
All that sound reasonable?

Very good point.

Now for my real feelings. I think systemd is an abomination that must be stopped.

Ah, we share the same feelings.

I'm actually more afraid of Canonical's Snap packages, but that's a different issue.

Oh, snaps are cancer over cancer. Thankfully at this time they are just a nuisance and easy to remove from the server environment. I have no doubt in a few years they will become a bigger problem for Ubuntu because their developers won't stop, just like systemd folks won't.

OTOH, journald is a work of art.

I see no reason to run it while depriving myself of the decades of experience the world has invested in Unix command-line tools (awk/grep/etc). Text logs are comprehensible and searchable in countless ways. journalctl has counterintuitive things going on. IIRC, `journalctl -f` is not enough to follow logs, you have to ask it more nicely than that. I may be wrong on this one, but a problem of a similar vibe killed my desire to look into it further when I have tried to embrace it.

found it was all there, stored more efficiently, easier to find just the stuff I wanted, and very easy to setup a remote log server.

I fully agree with you that this might well be the case.

However journald, like the rest of systemd-* was a solution in search of a problem that didn't exist in the first place. Remote syslog was a thing for decades and it does the job.

Seems there are too many options for dependencies built in.

This was swollen to a point they had to come up with special tools to analyze and debug dependencies. Incomprehensible, opaque.

I'd add netplan yaml files to the list of "what the heck were they thinking" list.

That's my main rant: I genuinely don't understand why would they choose a solution that is so much worse than the existing one. And force it on everyone.

it is now a core requirement for any Ubuntu Server.

I'm not sure about that. See my linked script: netplan is easily removed and replaced with ifupdown package and it does work.

I'd like to avoid systemd-resolved and systemd-timed as well. Today, I manually purge those last 2 and install tools that actually work.

Those two packages were designed to be destroyed. Like a purposely left packaging wrap on the dashboard of a new car when delivered to the owner.

PulseAudio to become stable ... and now it is being replaced.

That's the problem with kids: once something work it means it's boring and boring has to be messed with. Yeah, I sound like a hardcore luddite, which I am definitely not at all. Again, I like the core systemd as PID1.

Hopefully, we don't have to wait so long for systemd to be yanked from distros

LP and the team are inevitably going to grow up and fingers crossed systemd will be abandoned as well. Fingers crossed.

I don't know if you feel better, but I do. Thanks!

Thank you!