• LOGIN
  • No products in the cart.

Problems with Systemd and Why I like BSD Init by Randy Westlund

In my first post on this blog, I said that systemd was one of the things that pushed me to look into the BSDs and eventually move everything to FreeBSD. People have asked me what my problem with systemd is, but I haven’t written about it. So finally, here are my thoughts.

I’ve been reluctant to discuss systemd there because I was sick of everyone fighting over it. The Gentoo mailing list (and everyone else on the internet) became a giant flame war where everyone either thinks that systemd is evil and attacks Lennart Poettering personally, or thinks that systemd is going to save us all from ourselves and everyone must use it. I was just sick of everyone being so unreasonable.

For my part, I’m not a fan of systemd but I also don’t think it’s the end of the world. I watched a great interview with Lennart on the Linux Action Showabout why he implemented it, and he had some good reasons. To write a daemon for Linux, you need to maintain a different init script for each distro because they all put things in different places. And sysVinit isn’t the best with dependencies. Developing things for the Linux desktop is not as easy as it could be, due in large part to the fragmentation.

I avoided it when I was running Linux at home, but these days I have to use Linux for work and can’t avoid it any longer. After using systemd for a while, there are definitely some things I like about it. It makes easy things easier. Like taking some daemon I wrote and making it a service. The service files are nice, and telling it keep restarting my service if it crashes is super handy. I like that the service file will work on any Linux distro with systemd. I can just about write a service file from memory, whereas with an rc script I’ll always need to start from an example.

On the other hand, it makes the hard things even harder. If I want to do something complex that systemd doesn’t have an option for, I’m out of luck. With BSD init, I can just edit the /etc/rc script and replace it with arbitrary shell code. By default it looks under /etc/rc.d/ and runs the enabled startup scripts there, but I can change that in less than five minutes.

Here’s FreeBSD’s /etc/rc. /sbin/init calls this script, and /etc/rc calls all the enabled scripts under /etc/rc.d/.

https://github.com/freebsd/freebsd/blob/master/etc/rc

Notice how tiny it is. I can change anything in here I want. I can remove the part where it starts services and manually start just one or two services. I can change where the console output goes, like redirecting it to a file. I can make this script call whatever I like, however I like. Unlimited flexibility.

I’m not an expert in init systems, but I used OpenRC for a long time and I’m pretty familiar with BSD init now. From what I understand, BSD init is superior to the old sysVinit that Linux used to use. And Gentoo’s OpenRC is similar to BSD init. I don’t have much experience with classic sysVinit. So that’s where I’m coming from.

Systemd is very opaque. What I mean by that is that I can’t easily debug it. I can’t look inside and see what it’s doing, because it’s a complicated binary. I can’t easily make changes to the way it behaves. BSD init is just a script, so I can look at it, add print statements, and change it in any way I like. It’s easy to look at it and understand exactly what it’s going to do and why.

The way systemd does parallel startup bothers me. If service A depends on service B, they were traditionally started in sequence; A only starts after B is done starting. Systemd will start them both at the same time, but buffer any messages A tries to send to B until B is ready. So A just thinks that B is being slow to respond, and system startup is faster. You can disable this for certain things by using the ‘After’ directive, but that doesn’t apply to the whole system and anyhow, disabling a feature isn’t an answer to whether it should exist in the first place.

My problem with this is that the order in which services are started should, in my opinion, be exactly the same each time and predictable to the sysadmin. With systemd, the order is not deterministic, so you don’t know what’s going to happen next time you boot. I work with servers and embedded devices; I don’t care much about boot time. A server spends several minutes in the BIOS during POST anyway, before the bootloader is even run; making the OS boot faster doesn’t change very much. Embedded devices already start quickly because you trim them down to the bare minimum. What I care about is that every time I boot, the same exact things happen in the same exact order — the order that I want.

It seems no one can agree on whether systemd is modular or not. I think the problem is with different definitions of ‘modularity’. Systemd doesn’t put everything in PID 1 like some people suggest; it uses modules that communicate with each other. So it is modular in that sense. But these modules are very tightly integrated. You can’t easy remove some of them, or replace them with other things. So in that sense it is very monolithic. This is not at all like having a simple interface and passing data via stdin and stdout, which is the modularity that makes UNIX pipes possible. This is the sense that matters to me.

I dislike the way systemd is absorbing everything. It’s not just an init system, it’s become an everything-under-the-hood includes-the-kitchen-sink management system. That doesn’t feel modular to me. Why should systemd implement NTP when ntpd already exists? I think systemd-timesyncd and all the others like it are just reinventing the wheel.

In the case of ntpd, you might say that it’s a monstrosity that needs to be replaced with something much smaller, and you’d be right. There are others re-implementing it. OpenBSD created OpenNTPD, which is very lightweight, client-only, and can even hit HTTPS sites to verify that the times it gets from NTP servers is reasonable. Cleaning up or re-implementing ntpd is a good thing, but OpenBSD didn’t feel the need to make it part of init.

Linux is very fragmented and systemd is bringing things back together a bit. That may be a good thing. But the way we’re headed, there will come a point when we won’t be able to say ‘GNU/Linux’ anymore. It’ll be ‘GNU/systemd-linuxd’.

But part of what has made Linux so special is the ability to swap out literally any component on the system with something else — even libc! All a Linux program needs is a kernel and something that looks like a libstdc.so. It’s common to build embedded Linux devices with ulibc, or something else tiny. BSD can’t just replace its libc, it’s too integrated.

There was a really good interview with some GNOME devs on BSD Now not long ago. They said they were going to pay more attention to BSD compatibility, but also talked about why they needed to depend on systemd: There is no API or standard way to do things like set the timezone, change wifi networks, or perform many common desktop tasks on Linux. GNOME’s code is full of #ifdefs so that they can use different code for each distro or OS, and it’s not maintainable. Systemd provides these APIs in a clean and cross-platform manner, which is a big win for desktop environments.

OpenBSD is working on systemBSD, which is a systemd shim. It provides the APIs that GNOME needs, but doesn’t implement the rest of systemd. I think having APIs like this is a really good thing, and I’d like to see us as an industry move toward standardizing on it (or something like it).

Systemd still feels unstable. I’m not sure whether it’s because it actually has more problems than other software or just because people are quick to publicize its flaws, but it’s had some problems that would be really funny if they weren’t so serious. I’ve included links to the actual bug reports:

For a critical piece of infrastructure, having regular bugs like this is a big problem. It renders machines unbootable and ruins people’s days. Compare this to a really well-managed project like OpenZFS. To my knowledge, no one has ever lost data as a result of a bug in ZFS. Infrastructure projects like this have to be held to a higher standard than most software because the consequences for a bug can be so severe.

I think the systemd devs suffer from NIH syndrome. I can’t find anything to suggest that they considered OpenRC; they seem to have a personal problem with Gentoo. Many of the problems they described with init systems are already solved by both OpenRC and BSD init. Now they’re going around re-implementing every possible Linux daemon as a systemd module, just because they can.

I was talking to a reader recently who said, “Every time I look at systemd development, I get the feeling that it’s all a big rush to go somewhere.” I think that’s very true. Systemd was forced on the community, and it all happened very quickly. There was a lot of political maneuvering and strong-arming of distro maintainers to make it the default everywhere before it was mature.

Debian should not have adopted it for at least a few more years; they’re supposed to be the slow, steady, and stable distro. Their quick move to systemd hurt a lot of feelings and caused half their team to leave for Devuan. That shouldn’t happen. If your team is that fiercely split on an issue, the correct response it to leave the status quo alone until cooler heads prevail. Debian lost a lot of their reputation for stability because of this.

Honestly, I think if systemd had of been gradually adopted rather than forced, it would be a lot more popular. People could have seen Fedora and Arch use it for a few years and then maybe everyone would see the advantages and want to adopt it once it had proven stable. But the way it was, most of the community had no voice in the matter, and that’s not a good thing.

I would have welcomed a sysVinit replacement that was more UNIXy. Something like systemd would have been nice, but systemd has too many flaws in both design and implementation to be what we needed.

I guess the silver lining is that it showed everyone the problems with the benevolent dictator model that Linux uses. I think people have been fleeing to the BSDs for the difference in community just as much as the technical superiority.

I find systemd’s lack of faith in UNIX disturbing. So come join the BSD community. We have cookies 😉


publicpreview-1.phpBio: I’m an engineer and open source advocate living near Cambridge, MA. I do a lot
of freelance work and work from home most days.

My areas of focus include electronics, PCB design, microcontrollers, robotics,
Linux, BSD, system administration, web design, and RF communications.

I’ve been to Antarctica twice, doing satellite communication work for NASA.

I’m a licensed amateur radio operator, callsign KK4DOP.

In my spare time, I enjoy hiking, photography, and running BSD servers on
computers other people throw away.

Source TextPlain

Twitter @randy_westlund

LinkedIn Randy Westlund


Let us know, what do you think about the article in the comments down below!

October 17, 2015

17 responses on "Problems with Systemd and Why I like BSD Init by Randy Westlund"

  1. Good post. It pretty much sums up how I feel about systemd. Everyone turning it up to 11 isn’t very productive when discussing.

  2. Hmm. Nice article. I also really like the transparency that new-style BSD init has. I recently did a diff for /etc for a few of our FreeBSD servers and (with ZFS) they essentially have the same /etc apart from rc.conf.local. Which is … somehow awesome.

    As a little bit of criticism, however: FreeBSD’s /etc/rc is tiny, but it pulls in /etc/rc.subr, so the set of init scripts as a whole is really not that tiny a codebase.

    Of course, /etc/rc.subr is only a library of shell functions and variables that all init scripts you can assume to be set and available.

    Which is –from my personal point of engineering– exactly what systemd gets wrong: It gives you a comfortable box and when that box is all you need, you are fine. To be fair, they also do a great deal to make sure the box is as comfortable as they can make it. However, stepping outside of the box can become very, very painful. Because in the end, you need patch a very long C function deep inside systemd (and yes, I actually did that for an experiment).

    There’s a different way: You can still give the user a comfortable box It should be comfortable for most tasks, but –and that’s the idea of an extensible system– if you want something special, you can also step out of the box without too much pain.

  3. Curious how this article doesn’t mention Snowden. It’s well-known that RedHat works closely with the NSA. You can call it wild speculation. Not-so-wild after Snowden. I guess it was bound to happen as linux became corporate and people started chasing “market share”, a ridiculous concept unless you happen to be corporate too.

  4. I think Systemd is one of the NSA projects that they use to compromize OS’s on a large scale.
    Pushing it everywhere, even against all common sense, is the symptom. The clusterfsck that Systemd has become is good enough proof of it in my books. In the history of UNIXes, this same development has applied to all the critical system components like Bind, Sendmail, etc. that NSA has considered critical to be always exploitable and pwnable, for USA’s national security. We all end up paying the price.

  5. I run large databases on Oracle Linux 7[.2], and I don’t have stability problems (aforementioned bugs notwithstanding).

    I might point out that busybox includes an init that is easy to use with systemd (as well as an ntpd, which likely isn’t).

    I have previously documented an nspawn of thousands of busybox containers – System V init obviously cannot do this:

    http://www.linuxjournal.com/content/infinite-busybox-systemd

  6. Having primarily come from RedHat-land, I was always somewhat confused about what problems people seemed to be having with SysV init scripts. On the rare occasions where I needed to create one from scratch, I’d usually head over to the Fedora packaging page (or EPEL, now), cut and paste from https://fedoraproject.org/wiki/EPEL:SysVInitScript#Initscript_template , and customize as needed. What’s so hard about that?

    Having gotten more experience with Debian recently (and older Ubuntu), I see a little more of the pain people were experiencing. That being said, the pain seems to come from the distro and not from anything inherently wrong with SysV, init scripts, or shell generally. Seriously, shell scripts aren’t rocket science.

    The primary problem for me is accessibility. At 3am on a conference bridge to deal with a production outage, I want to be able to control the logical execution of as much of the system as possible. SysV, by handing off from /sbin/init to /etc/rc as quickly as possible, lets me do exactly that. Any competent SysAdmin should be able to understand the flow and modify as needed to solve a problem. This isn’t the case with systemd, which restricts your logic control to merely those options that the creators have provided. Need something more? Recompile it. Whatever increase in boot performance that systemd can provide over SysV/upstart or DashAsBinSh is nowhere NEAR enough justification for that level of lockout.

    The second major issue is the land-grab. The pattern behind systemd’s growth from humble upstart replacement proposal in Fedora 14 to the monstrosity it is now should be blatantly familiar to anyone who’s worked in a larger corporation, dealt with team politics, or paid attention to the tech industry for the last 20 years: embrace and extend; increase the cost of switching away; grab land and responsibility; add ‘convenient’ defaults; undersell true intent; impose vision. The fact that LP, KS, and the rest of the systemd/freedesktop cabal continue to deny that that’s, in fact, been their MO says more about their behavior than anything else.

    I expected better from the old-hands on the Fedora distribution list. I suppose the SA greybeards were too busy handling production issues with EL and its derivatives to pay attention to the goings-on. One day we all woke up and the Fedora heads had turned into the next generation of developers, convinced they could change the world by abjectly failing to learn any of the lessons of history whatsoever.

  7. Well put CorporaCallosa, and you too Randy. I too run a physical server with RAID-10 array that on a reboot takes 3-5 minutes JUST to POST and enumerate all the subsystems and backplanes, BEFORE it even starts to boot.. So I don’t give a flying flip about shaving 10 seconds off boot times. And since I don’t allow local users, keep my network exposed services all patches, I only reboot when there’s a remote-execute attack vector reported anyway (once or so a year).. soooo.. again.. reboot-speed means squat to me.

    So I’m an admittedly “old school” Linux sys-admin, but in my job I’ve sucked it up learned how to use systemd. I’m no systemd “expert”.. and so it’s been a little clunky getting up to speed.. but I’m more comfortable with being responsible for a systemd system now (run two or three CentOS7 systems now). However, the loss of that transparency, actual control and UN*X-power are my functional pains around systemd (that and the new stability problems I’ve seen on my systems too. some daemons can’t even be restarted and must be killed or the system rebooted!). Systems issues aside and on more of the tin-foil-hat side.. there is much more to be concerned about. You just don’t see scope creep like this in ANY other part of any open source OS. This combined with the other aforementioned governmental/political concerns (in other parts of this thread) creates a real market-share risk issue for systemD based distros (whether real or imagined, it IS affecting market share right now, and is quietly coming to a head right now).

    Anyway, like many of you, I too am in a holding pattern right now.. I have some friends in the industry who are looking into some of my own concerns.. but I am no longer evangelizing systemd based distros and instead (for those who ask) are recommending people check out Linux Mint (with the systemd shim) or check our a more open, less opaque BSD.

    This really is a shame.. I really really love Linux.. I have for 20 years now. I will hate to move away from Linux, but I feel there really may be no other choice here.. And choice, control and self determination is what Linux/FOSS has always been about. The exact opposite of systemd.

    Cheers all,

    Tweeks

  8. For a modern linux without systemd, look at: VoidLinux

  9. Systemd-journald using 100% CPU issue can be easily fixed. The only case where is may use high CPU is when coredumps are saved in the journal. Just set ‘Storage=none’ in ‘/etc/systemd/coredump.conf’ and compile systemd with elfutils support enabled. backtraces will be logged instead and the CPU issue will be fixed.
    journal corruption is real. It’s bad even though corrupted journals are rotated and eventually automatically deleted.
    Most Linux users wouldn’t even notice what init system they are using. You were already using udev.
    Systemd is pure Linuxism. Obviously BSD is going to feel alienated.

    • Hussam Al-Tayeb says:

      Just set ‘Storage=none’ in ‘/etc/systemd/coredump.conf’ and compile systemd with elfutils support enabled

      Well, that’s a blocker right there. I shouldn’t have to recompile my init system in a distribution, period. I shouldn’t have to edit /etc/rc either, but that’s at least administratable as a shell script… If /bin/init needs to be recompiled, there’s something very, very wrong going on.

Leave a Message

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© HAKIN9 MEDIA SP. Z O.O. SP. K. 2013