I have been working on Movim for many years now but I never had really the opportunity to explain how Movim is working. Let's take some time to write that :)
This will be split in several articles. I'll start with the big picture and slowly go more inside the project to explain how things are working on a lower level.
I am using social platforms for a long time and I am still surprised to see that even with the billions of dollars invested by the Internet giants they all still works pretty much the same way architecturally wise.
Even with all the new exciting technologies that we had the past few years things still looks the same to me: you publish content using Ajax calls (or through a Websocket), it is saved in the database and your contacts will ping the server once in a while to see if there is something new to grab.
This is quite convenient to built and if you want to do it yourself most of the frameworks out there are offering all the tools to do that easily. You install something like Laravel, Symphony or Zend (sorry people I'm more a PHP guy), put a SQL database in the back, add a couple of REST endpoints a lovely frontend and boom you have your social platform.
But why having to wait to get the publications, why having to pull things all the time?
For decades now we have had chat technologies that allows us to send, in real time, content across the globe, without any trouble. Why can't we do the same things for social stuff?
And here is the core idea of what Movim is made of.
Let's build a (real) real-time social network.
So, one of the mistakes that I tried to avoid when the project started was to reinvent the wheel. You will see that Movim is mostly made of basic and already proven technologies put together.
To built a real time social network I needed to transfer content instantaneously across a network. The content needs to be transported in connected (statefull) systems. It will then be socket based (as opposed to request based systems).
This pretty much excludes already all the new shiny social technologies standardized by the W3C (bye bye ActivityPub and WebSub). Those are built on HTTP. I'm not blaming those standards, they are perfectly valid and seriously defined solutions but it is not what I needed for Movim.
I needed a protocol that was:
- Real-time based
- Standard (means, with RFC and other documents that I can build on)
- Preferably already battle tested
- Widely deployed
Basically XMPP brings several advantages here:
- It is real-time (yay!), so basically all the communications are actually XML (yup, I already see you coming JSON people, but let's not dive into that kind of argument) packets (called stanzas here) sent trough TCP pipes with some TLS around.
- XMPP is offering a really basic and generic framework with many (many!) extensions that you can pick to build the solution you want on it. Those are the ones that I took to built Movim for example. And because it is XML based you can pretty much extend what you have by embedding existing namespaces. You take Atom you add Pubsub and boom you have a full real-time publication system for articles and attachments already fully defined and specified.
- You can do way more than simple "social" stuff with it. So no need to compose with 10 other protocols, XMPP is already offering everything to do chat and chatrooms, video-conferencing, publish-subscribe solutions and many other things. This has the advantage to keep the code quite concise because you are only talking one protocol in the backend.
- XMPP is federated, using a network a bit similar as the email one. The accounts are created on the servers (so that is why the identifiers are also similar: email@example.com). Clients are then connecting to those XMPP servers. On top of that you can have several clients connected, at the same time, to your account and they will all be synchronized in real-time :)
- There is already a big community, with serious servers out there that can handle millions of connections simultaneously without any issues (ejabberd <3).
This is also a big advantage for Movim. I don't have to take care of all the network issues. It is "just" a simple and dumb client that connects to XMPP servers and gets/sends content to them.
If you compare also this solution with other federated network solutions such as Mastodon or Diaspora this is also a big difference; for Movim the accounts are actually sitting on a distinct server. So I don't have the need to create another API to communicate with Movim. Everyone can exchange with Movim by just implementing XMPP (and there is already many libraries and solution in the wild to do so).
Ok, we chose the protocol, now we need to build the backend.
A bit of history
It is interesting to explain a bit the history of the project at this point. Movim was created in 2008 as a little experiment to learn programming and to try to develop at the same time a social platform from scratch. I took PHP as a base to build the backend and it was not intended to be more than a simple website with a few social features built-in.
During my studies and with a lot of self learning I improved progressively the project with the precious help of an another friend that left the project a couple of years later (Etenil, if you read me).
In 2014 I finally decided to move to a full real-time solution. The existing one was still based on some real-time "emulation" on top of HTTP to connect to the XMPP network (using the BOSH extension). This rewrite came as well with a huge refactoring of the internal architecture and a redesign of the user interface. But… I kept PHP because I had already a lot of knowledge in this language and I knew already the limits of developing with it. I also had a large part of the codebase (mostly the whole XMPP part) that could be directly ported to the new architecture with only a few adjustments.
At this time a new project was also emerging in the PHP community, ReactPHP. This framework was specifically designed to handle real-time architectures in PHP.
I decided to give it a try.
Launch a daemon, connect some pipes, et voilà!
ReactPHP is coming with a lot of side projects that allows to create all sorts of real-time architectures.
At this time in the project I'm using:
- react/event-loop the core loop that handles all the I/O
- react/dns an asynchronous DNS resolver
- react/promise-timer to fire some events once in a while
- react/socket to connect to XMPP using pure TCP sockets (with TLS on top)
- react/child-process to launch and control sub processes
- react/stream to connect all those things together
- react/zmq a ZeroMQ binder to have proper messaging system between processes
After a few months of experiments I came up with an architecture that hasn't really changed since then.
Let's have a look more precisely how this is working.
All the Movim structure is handled by one central daemon (called
daemon.php, amaze!). This daemon is handling all the Websockets of the users browsers (mobiles and desktop) and is launching a sub process
for each user connected. Then it is only acting as a dumb router that forward messages between the user Websockets and their respective processes.
This architecture is bringing a few advantages:
- The user sessions are isolated and are not affecting each others performances wise
- They can more easily be controlled (killing one session will not bring down all the others)
- The main daemon is minimalistic (basically acting as a dumb router)
And has one disadvantage:
- The memory consumption is higher. The code is loaded several time between all the sub-processes. So count for ~10 to 20Mb per connected user. This is indeed a problem that was greatly improved the past versions by reducing the memory consumption of Movim itself, cutting out some dependencies and also moving to more recent versions of PHP itself (PHP 7.0+) but is still a challenge for the upcoming versions. This also bring a scalability issue. The backend can easily handle thousands of connections at a time but you'll run out of memory before reaching that point.
Each of those sub-processes resolves and connects to the user XMPP server and then handles all the communications done with it. They also connect to a common SQL database that acts as a caching layer for each account but also to share data between them (to allow the discovery of public resources for example).
Finally those processes also handle all the frontend related things, but this will be detailed more precisely in an upcoming article.
Some optimizations were made then to improve the overall performances of this architecture. Here are the 3 main ones that I think had the biggest impacts on the project.
XML stream parser
An XMPP connection is basically a bidirectional stream of XML. The client is sending XML requests (called stanzas) and is parsing the incoming ones. One interesting thing about the incoming ones is that they are all part of the same XML "document". Initialy Movim was detecting each of those stanzas and parse separately.
The parser was then rewritten to work as a stream (see PHP XML Parser). This allows Movim to already prepare the incoming stanzas and fire the related events as soon as the first XML elements are received inside the socket.
This small change really improved the overall XMPP performances of the project. Especially during the connection phase, Movim can now handle several thousands of stanzas in a couple of seconds.
The communications between the main daemon and the sub-processes (called "linkers" in Movim) were initially done using simply stdin/stout. This brought some buffer and performances issues and I choose to use something designed especially for that: ZeroMQ.
Basically, each time the main daemon is launching a linker for a user it also creates two dedicated IPC streams (one for incoming, one for outgoing messages) and then handle everything that is going through them.
This allowed me to remove some buffers that were used to pass those messages along and boosts the overall performances, especially for the UI part.
ZeroMQ is also really lightweight and is already available and packaged in many GNU/Linux distributions.
From Modl to Eloquent
Movim was relying initially on a specifically designed database library (called Modl). The upcoming version (0.14) will ship with the known Eloquent library (used in the Laravel framework).
This change also allowed me to use some fancy features such as eager loading, lazy loading and do proper migrations in the database to boost some requests.
More info of the dedicated post here.
What did I learn?
Through all those years working on this real-time project I can now make a couple of conclusions regarding the choices and changes that I made:
- PHP is not a problem most of the time: PHP is fast, really fast. Most of the optimizations that I made were related to the way I was handling the streams and their contents and the requests in the databases. PHP7 and the following versions did improved slightly the performances but it was minor regarding the other changes made in the codebase.
- Check what is blocking: when you are working in real-time, even if things are handled by promises and other asynchronous systems you will have blocking code. Ensure that this code is not "too slow". In Movim for example, the database requests are still considered as blocking (this is another optimization that can be done…) so if a request is taking 200ms to be triggered, it can push back the execution of some code of 200ms.
- Do some proper "real conditions" testings. I thought Movim was fast until I saw some users struggling with it (Nik, if you read me). Some of the Movim users had way more chatrooms and subscribed feeds than I expected which created some big slowdown, especially during the connection phase. With some proper (and sometimes really simple) optimizations things were brought back to normal.
And maybe the most important one: Keep It Simple!
I have the feeling that a lot of projects are jumping into the DRY (Don't Repeat Yourself) principle a bit too much. Sometime you don't need to import a full library, just write the function that you need and move forward, try to have the less dependencies possible.
Always check what are the requirements of your project and always question their necessity.
Finally don't be afraid of big refactorings (it took me 50 hours of work to move from Modl to Eloquent) to simplify and cleanup your code base if it's necessary.
So is Movim fast?
Movim is fast. In some cases Movim is even faster than some native XMPP clients such as Pidgin and Gajim. It is also faster then several other chat platforms, because of its backend, but also because of the way the frontent is designed. We will talk about that in an upcoming article.
On my account (400 contacts, 50 chatrooms) it only takes a couple of seconds to authenticate and have a fully ready and reactive UI, knowing that the data is coming from a third party (your XMPP server) and are re-synchronized for some of them when you authenticate.
If you want to look for yourself, you can try it out on our official website ;)
That's all folks!
We first worked on cleaning up and stabilizing the #dependencies of Movim. The outdated heyupdate/emoji was replaced by a wonderful pull request by mirabilos that add support of emojis directly inside Movim.
On my side I replaced ramsey/uuid with a simple internal function and worked on upgrating reactphp/http to their latest release with the help of WyrilHaximus, which also helped to release the v0.4.0 of reactphp/zmq that contains some important fixes for Movim.
The template engine of Movim, RainTpl was also stabilized to the latest release.
Natureshadow also made a really nice pull request to prepare the Debian package and fixe a couple of small bugs regarding URL handling inside the project.
All those dependencies will soon be packaged and integrated in Debian.
On top of that I worked a few hours yesterday on the optimisation of the #database requests by using some memory caching and Eloquent eager loading to prefetch some extra information when querying resources in the DB. This reduces the time spent to generate the pages and contents by more than 50 to 75% in some cases! It can especially be noticed on the Chat page and Contacts page.
That's all folks!