openEngiadina: From ActivityPub to XMPP

The openEngiadina project is developing a platform for open local knowledge. With regard to technology, this has so far consisted of:

• Research into a suitable data model:

  • How to use RDF for various types of data (including ActivityStreams for social interactions)
  • How to make RDF content-addressable
  • A scheme for robust content-addressing
  • How to cryptographically sign content-addressed content
  • How to use CRDTs for mutability (this work has been continued as part of the DREAM project)
• Development of a client for creating and curating content: GeoPub
• Development of a server capable of handling of such diverse types of content: CPub

The network protocol we have been using is ActivityPub. This seemed to be a perfect fit. We believe that managing knowledge is a social activity and ActivityPub is primarily a protocol for social interactions. Furthermore, ActivityPub can be extended to allow arbitrary RDF content via the JSON-LD serialization.

Unfortunately, we have come to realize that using ActivityPub is considerably harder than we expected:

• Using JSON-LD as an RDF serialization is very complicated. It requires the usage of algorithms (e.g. the Expansion Algorithm or the Framing Algorithm) that are quite difficult to understand and implement. This complexity seems to come from the desire to bridge Linked Data ideas to JSON in such a way that people familiar with JSON are not intimidated. For people who care more about the Semantic Web aspects and are not tied to using JSON, the additional complexity of having to use the JSON-LD algorithms is considerable. JSON-LD maybe was really just not intended to be an RDF serialization and trying to use it as such is painful.
• There are not many implementations of the ActivityPub Client-to-Server protocol (C2S). This made developing and testing the client and server more time-consuming as we had to develop the protocol in lockstep on client and server. At the end we were still only compatible with our own software.
• ActivityPub is not a complete specification and many additional protocols need to be implemented (e.g. WebFinger) in specific ways in order to be compatible with existing servers.

Over the last couple of months we have been exploring XMPP as an alternative to ActivityPub and are happy to announce an initial proof-of-concept version of GeoPub - the openEngiadina client - using XMPP. You can try it here (an XMPP account is required). The source code is available on Codeberg.

In the following I'd like to outline how XMPP works in the context of openEngiadina and sketch the next steps we are thinking of.

Update (2021-11-23): Changed wording in paragraph on JSON-LD to prevent it from being missunderstood as an argumentum ad hominem.

Update (2021-12-02): Changed wording in paragraph about there being practically no implementation of C2S protocol. There are a few. See this post on SocialHub.

XMPP for Open Local Knowledge

XMPP (Extensible Messaging and Presence Protocol) is a protocol for instant messaging but as the name indicates it is extensible and can be used for way more applications.

Similar to ActivityPub, XMPP is a federated protocol where clients connect to servers of their choice that interoperate and can exchange messages.

The core protocol is defined in an IETF RFC: RFC 6120: Extensible Messaging and Presence Protocol (XMPP): Core. Another document describes how the core protocol can be used for instant messaging: RFC 6121: Extensible Messaging and Presence Protocol (XMPP): Instant Messaging and Presence.

Further extensions are maintained by the XMPP Standards Foundation as XEPs (XMPP Extension Protocols). There are currently over 400 XEPs describing various extensions. However, it is not necessary to implement all XEPs in order to have a working XMPP client/server. The core RFC is enough to start communicating and with a handful of XEPs you can build a working application. The Modern XMPP project maintains recommendations for which XEPs to implement for which use-cases.

Publish-Subscribe

XMPP is most widely used for instant messaging (as in chatting) but it can be used to develop way more types of applications via the publish-subscribe pattern as described by XEP-0060: Publish-Subscribe. This allows an entity to publish content to a publish-subscribe node, other entities that have previously subscribed to the node will receive the content.

In ActivityPub terms this is similar to following an actor and then receiving activities that the actor creates.

The publish-subscribe (pubsub in the following) mechanism in XMPP is more flexible as XMPP pubsub nodes are not bound to users (or actors in ActivityPub). Nodes can be created, configured and deleted dynamically by users for specific content. For example you can create a dedicated pubsub node for notices of lost cats in a specific area.

Personal Eventing Protocol

The flexibility of being able to create pubsub nodes makes discover-ability a bit more tricky. When you create a new XMPP account you may know the contact details of some people and want to follow content they post. How do you find the pubsub nodes that these people have created to post their personal content?

XEP-0163: Personal Eventing Protocol allows XMPP users to themselves become pubsub nodes (in fact many name-spaced nodes). This makes discovering personal content much easier. It makes following people as easy as in ActivityPub.

Microblogging

For microblogging (i.e. small textual contents) there is an additional XEP (XEP-0277: Microblogging over XMPP) that defines the format of the content and the named pubsub node where such content should be published.

GeoPub support XEP-0277 and can be used to create posts with geographic location. This makes GeoPub compatible with Movim and Libervia - two projects that use XMPP for social networking.

The format for microblogging posts defined in XEP-0277 is based on the widely used Atom Syndication Format (as defined by the IETF RFC 4278). This allows content created on an XMPP pubsub node to be exported and made available as an Atom feed accessible over simple HTTP.

XML

XMPP is an XML based protocol. Messages and content transferred over XMPP is formatted as XML. This allows formats such as Atom, XHTML, Docbook or RDF/XML to be directly transported over XMPP.

For openEngiadina we intend to use RDF/XML (an XML serialization of RDF) over XMPP. In fact, we intend to use the ActivityStreams vocabulary serialized as XML over XMPP. This allows, at least in theory, a very clean interoperability with the existing ActivityPub network.

The holy grail of ActivityPub servers is a generic server capable of handling any kind of data. With XMPP this is normal. XMPP servers transport XML but do not care about the content itself. Any existing XMPP server is adequate for openEngiadina.

Messaging and End-to-End Encryption

Another advantage of XMPP is that instant messaging is supported directly in the protocol.

A core tenet of openEngiadina is that content creation and curation is first and foremost a social activity. Quick and direct messaging is an important aspect of it and having this functionality built-in to the protocol is extremely valuable.

Furthermore, XMPP supports secure end-to-end encrypted communication (via XEP-0384: OMEMO Encryption).

Existing Software and Related Projects

One major appeal of XMPP is the availability of high-quality implementations (servers, clients and libraries).

On the server side there are ejabberd and Prosody. Both are well-tested and robust pieces of software that work and can be relied on.

XMPP clients exist for many platforms (see this list). There has been a renewed drive to develop and polish clients in order to make them viable alternatives to commercial instant messaging services (e.g. Dino, Snikket). Other clients are already extremely usable and mature (e.g. Conversations).

Why not Matrix?

Matrix is another federated protocol for instant messaging that also supports more complex applications.

Matrix does not seem like a good match for openEngiadina as the protocol is still in heavy development, there are practically no alternative server implementations. Furthermore, the protocol seems to do too much. It seems to be more like a distributed JSON database than a messaging protocol. This comes with a performance penalty and an inability to run Matrix servers on low-powered machines which is not acceptable for openEngiadina.

Other Changes

Previous versions of GeoPub were implemented in ClojureScript. GeoPub is now implemented in OCaml using the js_of_ocaml compiler. Reasons for this change were licensing issues as well as difficulties in providing reproducible development environments. A suitable and reproducible development environment for GeoPub can now be provisioned with Guix.

Hosting of code has moved from Gitlab to Codeberg. Reasons include that Gitlab stopped working properly with my browser and that codeberg seems to be a nice project.

Next Steps

Moving to XMPP is a bit of a step back as we need to re-implement all the research done towards the data model. This is what we intend to do next. In particular we intend to use content-addressed ActivityStreams over XMPP.

The fate of CPub - the generic ActivityPub server - is still undecided. We might re-purpose it for an XMPP to ActivityPub bridge or continue development as is.

Repression in Belarus

My fellow openEngiadina developer rustra is still being detained in Belarus on charges of organizing protests last year. He has reported being tortured.

This is a strong reminder of the ongoing repression in Belarus.

Please consider supporting activists and victims of political repression by donating to the Anarchist Black Cross Belarus or other organizations such as Amnesty International.

Acknowledgments

Thank you for making it so far and thank you for your interest!

The openEngiadina projects is supported by the NLnet Foundation trough the NGI0 Discovery Fund.