The perils of federated protocols

This article brought to you by LWN subscribers

Subscribers to made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

By Jake Edge
May 18, 2016

The lure of "federation" for internet services is potent, since it allows disparate providers to interoperate and users to choose the provider that (most) meets their needs—or to become their own provider. Many of the longtime services, such as email, web serving, DNS, and others, are federated, but many of the newest services decidedly are not. That tension is playing out right now for the Signal open-source encrypted messaging and voice application from Open Whisper Systems (OWS) and others who would like to be able to federate with it.

Signal and LibreSignal

The Signal app for Android is available under the GPLv3, but it is not fully free in the eyes of some because it relies on the Google Cloud Messaging (GCM) service that is part of Google Play Services. That means that some amount of metadata (but not the contents of the encrypted messages) traverses Google's servers. For privacy reasons, some have changed the Signal app to eliminate that dependency, which is something that its license clearly allows, but they still want those changed apps to be able to communicate with the rest of the Signal-using world. That's where the problem starts.

A service like Signal relies upon servers as intermediaries—and servers are not free. If apps like LibreSignal—a fork of the Signal app that removes the GCM dependency—want to communicate with other Signal users, they must either use the same servers or run their own servers that are federated with those run by OWS. But that is not to be.

In a thread on the LibreSignal issue tracker, OWS developer Moxie Marlinspike stated that OWS did not want LibreSignal to use its servers (nor to use "Signal" as part of its name):

You're free to use our source code for whatever you would like under the terms of the license, but you're not entitled to use our name or the service that we run.

If you think running servers is difficult and expensive (you're right), ask yourself why you feel entitled for us to run them for your product.

One of the LibreSignal developers, Michel Le Bihan (posting as mimi89999), said that the project was willing to change the name, but wondered: "If I finance running a TextSecure server for LibreSignal, will you federate with us?" Marlinspike deemed that improbable: "It is unlikely that we will ever federate with any servers outside of our control again, it makes changes really difficult."

A few years back, the encrypted-messaging piece of Signal, TextSecure, was federated with the CyanogenMod servers so that users of that platform could send messages to TextSecure users on other platforms. That federation is no longer happening; Marlinspike explained the problems that federating caused:

It seriously degraded the UX [user experience] for our users and held us back in the development process at many times. I'd estimate that all told, we lost about 6 months to a year of progress. It's something we'll probably never do again, and has fully convinced me that federated protocols are a thing of the past in this world of ours.

Marlinspike expanded his thoughts on federated protocols in a blog post. The crux of the problem with federated protocols is that the entire ecosystem is moving too fast for services to support them. It restricts what the service provider can change because the existing features still need to be supported:

When someone recently asked me about federating an unrelated communication platform into the Signal network, I told them that I thought we'd be unlikely to ever federate with clients and servers we don't control. Their retort was "that's dumb, how far would the internet have gotten without interoperable protocols defined by 3rd parties?"

I thought about it. We got to the first production version of IP, and have been trying for the past 20 years to switch to a second production version of IP with limited success. We got to HTTP version 1.1 in 1997, and have been stuck there until now. Likewise, SMTP, IRC, DNS, XMPP, are all similarly frozen in time circa the late 1990s. To answer his question, that's how far the internet got. It got to the late 90s.

That has taken us pretty far, but it's undeniable that once you federate your protocol, it becomes very difficult to make changes. And right now, at the application level, things that stand still don't fare very well in a world where the ecosystem is moving.

Indeed, cannibalizing a federated application-layer protocol into a centralized service is almost a sure recipe for a successful consumer product today. It's what Slack did with IRC, what Facebook did with email, and what WhatsApp has done with XMPP. In each case, the federated service is stuck in time, while the centralized service is able to iterate into the modern world and beyond.

So while it's nice that I'm able to host my own email, that's also the reason why my email isn't end to end encrypted, and probably never will be. By contrast, WhatsApp was able to introduce end to end encryption to over a billion users with a single software update.

But he also recognized some of the downsides to his conclusions. Federation allows users to choose who has access to their metadata, but it generally already a lost cause because there are typically just a few providers—or even a single provider (e.g. Gmail)—that provide most users with the service. Though he believes it is impossible to have a new federated service on today's internet, he is not entirely happy with that outcome: "it's something that I'd love to be proven wrong about".


Various posters in the issue-tracker thread pointed to Extensible Messaging and Presence Protocol (XMPP) as a potential solution, along with various projects (such as Conversations and ChatSecure) that use XMPP. Marlinspike is not particularly hopeful that XMPP-based solutions will lead to a successful messaging network, as Signal's non-federated approach has done. He noted that the Guardian project has been working on the problem as long as OWS has, "so why are Signal's growth, ratings, and engagement substantially higher?"

In his blog post, he gets more specific about the shortcomings of XMPP that he sees. While it is an extensible protocol, that extensibility leads to problems of its own:

What we have instead is a complicated morass of XEPs [XMPP Extension Protocols] that aren't consistently applied anywhere. The implications of that are severe, because someone's choice to use an XMPP client or server that doesn't support video or some other arbitrary feature doesn't only effect them, it effects everyone who tries to communicate with them. It creates a climate of uncertainty, never knowing whether things will work or not. In the consumer space, fractured client support is often worse than no client support at all, because consistency is incredibly important for creating a compelling user experience.

One of the most user-friendly choices that Signal made was to use the phone numbers already stored in the contacts list as the identifiers for sending messages—exactly like regular SMS text messages. It is "not possible to build an identity this simple in a federated landscape", Marlinspike said. The focus for Signal is on making it usable for ordinary users:

We're trying to make mass surveillance impossible for the world we live in, not a fantasy land inhabited only by cryptonerds and moralists. This is the world we live in: people do most of their communication on mobile devices running iOS or Android, use Chrome on the desktop, and expect contact discovery to be automatic in their social apps. The browser has won the desktop, iOS and Android have won mobile, and the velocity of the ecosystem is unlikely to make "distributed" communication mechanisms possible for some time.

We want to produce technology that is privacy preserving but feels just like everything else people already use, not somehow convince everyone to fundamentally change their workflow and their expectations.

But the fact remains that Signal uses Google services and various people (including some that are precisely the target audience for encrypted messaging) do not trust Google or, perhaps, worry about what the company might be compelled to do by various governments. Those who want to communicate with other Signal users but not with Google servers are shut out. In a post on his blog, Matthew Garrett laments the situation:

This is awkward. Signal is deservedly popular. It provides strong security without being significantly more complicated than a traditional SMS client. In my social circle there's massively more users of Signal than any other security app. If I transition to a fork of Signal, I'm no longer able to securely communicate with them unless they also install the fork. If the aim is to make secure communication ubiquitous, that's kind of a problem.

Right now the choices I have for communicating with people I know are either convenient and secure but require non-free code (Signal), convenient and free but insecure (SMS) or secure and free but horribly inconvenient (gpg). Is there really no way for us to work as a community to develop something that's all three?

Comments on that post point to various options that may more or less fill those needs, but the lack of a network effect for projects that were listed, such as Matrix, make them a hard sell in the "convenience" department. But "walled gardens" are just that—federation is one way out of that particular trap. For companies that are trying to build their business, though, walled gardens have some obvious appeal, which may also be playing into the plans of OWS.


Part of the reason that Marlinspike's thoughts are striking a chord in some circles (this Hacker News thread for example) is because of his reputation in fields like cryptography and security, as well as for having been instrumental in building the popular Signal service and apps. Much of the internet is built on federated technology, which has led to a lot of innovation and important progress along the way. It is concerning to many (perhaps including Marlinspike) that centralized services may be the way forward.

It is a difficult problem. Free, secure, and convenient solutions in the messaging space have not (yet?) come about. Even non-realtime encrypted communication via email is inconvenient, at best, and effectively unusable by those who are not tech savvy. Trusting some centralized service to handle all that may provide convenience, but there are always going to be concerns about the trustworthiness of that provider (and the code it runs). This is not really a new problem for the free-software world, but Marlinspike's thoughts have brought it into sharp focus. Difficult or no, it is a problem worth solving.

(Log in to post comments)