Systemd as tragedy

Did you know...? is a subscriber-supported publication; we rely on subscribers to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the net.

Tragedy, according to Wikipedia, is "a form of drama based on human suffering that invokes an accompanying catharsis or pleasure in audiences". Benno Rice took his inspiration from that definition for his 2019 talk on the story of systemd which, he said, involves no shortage of suffering. His attempt to cast that story for the pleasure of his audience resulted in a sympathetic and nuanced look at a turbulent chapter in the history of the Linux system.

Rice was also influenced by Aurynn Shaw's writing on "contempt culture". According to Shaw, people use contempt (of developers using a different programming language, for example) as a social signifier, a way of showing that they belong to the correct group. This sort of contempt certainly plays into this story, where large groups identify themselves primarily by their disdain for systemd and those who work with it. A related concept is change, or the resistance thereto. The familiar is comfortable, but it isn't necessarily good, especially if it has been around for a long time.

The roots of the tragedy

The ancestry of systemd, he said, is tied to the origin of Unix, which was "a happy accident" — a reaction to the perceived complexity of the systems that came before. It was brutally simple in all regards, including how its user space was bootstrapped. Putting up an early init man page, he [Benno Rice] called out the "housekeeping functions" that it was designed to carry out, including mounting filesystems and starting daemons. Those are two distinct tasks, but they had been lumped together into this one process.

In those days, there were few daemons to worry about; cron, update (whose job was to write out the filesystem superblocks occasionally), and the init process itself listening on a few terminals were about it. By the time that 4BSD came around, Unix had gained a proper getty daemon, network daemons like routed and telnetd, and the "superdaemon" inetd. That is where things started to get interesting, but it still worked well enough for a while.

Then the Internet happened. Using inetd worked well enough for small amounts of traffic, but then the World Wide Web became popular and it was no longer possible to get away with forking a new process for every incoming connection. Sites on the net started running databases and other systems with a great deal of stored state that could not go away between connections. All this shifted the notion of a daemon toward "services", which are a different beast. Old-style init could start services, but was pretty much useless thereafter.

Part of the problem was the conflation of services and configuration. Tasks like mounting filesystems are of the latter variety; they are generally done once at boot time and forgotten thereafter. But that approach is not sufficient for automated service management, which requires ongoing attention. Thus we saw the birth of more service-oriented systems like Upstart and systemd. This is something other operating systems figured out a while back. Windows NT had a strong service model from the beginning, he said, and Mac OS has one now in the form of launchd. Other systems had to play a catch-up game to get there.

Apple's launchd showed up in the Tiger release and replaced a whole series of event-handling daemons, including init, cron, and inetd. Systemd, Rice said, was an attempt to take a lot of good ideas from launchd. When Lennart Poettering started thinking about the problem, he first looked at Upstart, which was an event-based system that was still based around scripts, but he concluded that he could do a better job. His "Rethinking PID 1" blog post cited launchd as an example to work from. He was concerned about improving boot speed and the need for the init system to be tuned into the hardware and software changes on a running system. When the Unix init system was designed, systems were static, but the environment in which the operating system runs now is far more dynamic.

The service layer

Classic Unix-like systems are split into two major components: the kernel and user space. But kernels have become more dynamic and changeable over time, responding to the hardware on which they run. That has led to the need for a new layer, the "service layer", to sit between the kernel and user space. This layer includes components like udev and Network Manager, but systemd seeks to provide a comprehensive service layer; that is why it has pulled in functionality like udev over time. It has been quite successful, achieving wide (but not universal) adoption through much of the Linux distribution space, often creating a great deal of acrimony in the process.

There are a number of often-heard arguments against systemd; one of those is that it violates the Unix philosophy. This argument, he said, seems to be predicated on the notion that systemd is a single, monolithic binary. That would indeed be silly, but that's not how systemd is structured. It is, instead, a lot of separate binaries maintained within a single project. As "a BSD person" (he is a former FreeBSD core-team member), Rice thought this pulling-together of related concepts makes sense. The result is not the bloated, monolithic system that some people seem to see in systemd.

Another frequently heard criticism is that systemd is buggy. "It's software" so of course it's buggy, he said. The notion that systemd has to be perfect, unlike any other system, raises the bar too high. At least systemd has reasonable failure modes much of the time, he said. Then, there is the recurrent complaint usually expressed as some form of "I can't stand Lennart Poettering". Rice declined to defend Poettering's approach to community interaction, but he also said that he had to admire Poettering's willpower and determination. Not everybody could have pushed through such a change.

Systemd makes no attempt to be portable to non-Linux systems, which leads to a separate class of complaints. If systemd becomes the standard, there is a risk that non-Linux operating systems will find themselves increasingly isolated. Many people would prefer that systemd stuck to interfaces that were portable across Unix systems, but Rice had a simple response for them: "Unix is dead". Once upon a time, Unix was an exercise in extreme portability that saw some real success. But now the world is "Linux and some rounding errors" (something that, as a FreeBSD person, he finds a little painful to say), and it makes no sense to stick to classic Unix interfaces. The current situation is "a pathological monoculture", and Linux can dictate the terms that the rest of the world must live by.

Systemd has gained a lot from this situation. For example, control groups are a highly capable and interesting mechanism for process management; it would be much harder to do the job without them. They are much more powerful and granular than FreeBSD jails, he said. Developers for systems like FreeBSD can see systemd's use of these mechanisms, and its subsequent non-portability, as a threat. But they can also use it as an excuse to feel just as liberated to pursue their own solutions to these problems.

Change and tragedy

The whole systemd battle, Rice said, comes down to a lot of disruptive change; that is where the tragedy comes in. Nerds have a complicated relationship to change; it's awesome when we are the ones creating the change, but it's untrustworthy when it comes from outside. Systemd represents that sort of externally imposed change that people find threatening. That is true even when the change isn't coming from developers like Poettering, who has shown little sympathy toward the people who have to deal with this change that has been imposed on them. That leads to knee-jerk reactions, but people need to step back and think about what they are doing. "Nobody needs to send Lennart Poettering death threats over a piece of software". Contempt is not cool.

Instead, it pays to think about this situation; why did systemd show up, and why is it important? What problem is it solving? One solution for people who don't like it is to create their own alternative; that is a good way to find out just how much fun that task is. Among other things, systemd shows how the next generation doesn't think about systems in the same way; they see things more in terms of APIs and containers, for example.

So what can we learn from systemd? One is that messaging transports are important. Systemd uses D-Bus heavily, which gives it a lot of flexibility. Rice is not a fan of D-Bus, but he is very much a fan of messaging systems. He has been pushing for BSD systems to develop a native message transport, preferably built into the kernel with more security than D-Bus offers. On top of that one can make a proper remote procedure call system, which is a way to make kernel and user-space components operate at the same level. In a properly designed system, a process can simply create an API request without having to worry about where that request will be handled.

Other lessons include the importance of supporting a proper service lifecycle without having to install additional service-management systems to get there. Service automation via APIs is important; systemd has provided much of what is needed there. Support for containers is also important; they provide a useful way to encapsulate applications.

Systemd, he concluded, fills in the service layer for contemporary Linux systems; it provides a good platform for service management, but certainly does not have to be the only implementation of such a layer. It provides a number of useful features, including painless user-level units, consistent device naming, and even the logging model is good, Rice said. Binary logs are not a bad thing as long as you have the tools to pull them apart. And systemd provides a new model of an application; rather than being a single binary, an application becomes a bunch of stuff encapsulated within some sort of container.

The world is changing around us, Rice said. We can either go with that change or try to resist it; one path is likely to be more rewarding than the other. He suggested that anybody who is critical of systemd should take some time to look more closely and try to find one thing within it that they like. Then, perhaps, the catharsis phase of the tragedy will be complete and we can move on.

A video of this talk is available on YouTube.

[Thanks to and the Linux Foundation for supporting my travel to the event.]
(Log in to post comments)