• Mo chevron_right

    From Modl to Eloquent, or how to change a social network database engine

    Timothée Jaussoin – pubsub.movim.eu / Movim – Tuesday, 17 April, 2018 - 08:53 edit

After 50 hours of work, more than 5000 line added and 10000 deleted, and 250 files changed it's finally done. This is the biggest refactoring done in the history of the project.

Movim is a social network and instant messaging web client for the XMPP network. It grab data from XMPP and store them temporary in a SQL database to present it in a light and simple UI to the users.

A Community on Movim, all the posts here are on XMPP and cached in the Movim database

The database library is therefore a really important piece of the architecture.

The idea was to replace the historical database layer (Modl) with Eloquent the reference database library of the Laravel project. This will bring many new interesting features to the project:

  • Have a more organized database with proper relations between the tables, have indexes and foreign keys (some of those features were missing in Modl)
  • Be able to do advanced queries to present more interesting content to the user
  • Simplify the code of the project (this is also why several thousands lines were removed from the code base)

Most of the existing features were ported to the Eloquent, some others are temporary disabled or simplified. Those, like the suggestions of Posts and Contacts, will be improved in the upcoming months.

You can also expect many small bugs here and there. The plan is to take a couple of weeks (months?) more to stabilize the code and do a proper 0.14 release.

For those that want to try out the current release, the Installation Tutorial on our wiki has been updated to reflect the changes introduced in this release. Here is a small sum-up.

Migrate from 0.13.* to 0.14+

Be sure to do a backup of your database before. It only takes a few seconds and it will allow you to do it again if you have any trouble during the migration. For PostgreSQL you can do.

pg_dump -d movim_db > movim_backup.sql

Then you can turn off the Movim service, if you are using systemd you can do.

service movim stop

Go to the Movim path and pull the new code, then update the Composer dependencies.

git pull
composer install

You don't have to change anything in the configuration, the daemon will also refuse to start if some of the following steps are not finished properly.

You can then migrate the database to create the new structure.

php vendor/bin/phinx migrate

Movim is also coming with a migration script. It will:

  • Move the existing users and their configurations to the new tables
  • Migrate the default Movim configuration
  • Migrate the database cache
  • Migrate the chatroom invitations script
  • Destroy all the old Modl table

You can call this script with the following command

php vendor/bin/phinx seed:run -s FromModlToEloquent

Once everything is done you can then restart the daemon :)

service movim start

Like always you can help us by covering our monthly costs on the Patreon page or by doing a one time donation on Paypal.

You can also help us by writing blog posts, tutorials or talk about the project on the other platforms (Reddit, Twitter, HackerNews…) :) If you have remarks or questions, do not hesitate to join us on the official chatroom, movim@conference.movim.eu.

That's all folks!

  • image
  • favorite

    11 Like

    valeg , Roelof Pieter , odomar , Adrien Dorsaz , debacle , Benjamin , Marzanna , Appiah , Johannes Brakensiek , Appiah , Appiah

  • Mo chevron_right

    Timothée Jaussoin – pubsub.movim.eu / Movim – Monday, 16 April, 2018 - 06:26

    Meetup in #Stockholm the 19th of April in the evening. Join us there to talk about Movim and related things :)

  • Stockholm Movim group

    This group is for anyone using Movim or curious to discover this social network."Movim (My Open Virtual Identity Manager) is a distributed social network built on top of XMPP, a popular open standards

  • favorite

    7 Like

    Timothée Jaussoin , arie , paulfree14 , Marzanna , Appiah , Appiah , debacle

  • Mo chevron_right

    A new release is coming, help up with the #translations

    Timothée Jaussoin – pubsub.movim.eu / Movim – Monday, 19 February, 2018 - 21:54 edit

The next #Movim version is coming soon, lots of nice improvements have been added since the Christmas #release, Movim 0.13 - Coggia.

Several new features were added those past few weeks like a picture picker when URL that contains several pictures are shared, a spoiler on NSFW content in the news feed, improvements in the video-conferencing feature but also code cleanup and stability improvements. A more detailled article will follow the new release, as usual :)

But before packing everything up, we need your help! We would like you to help us translating Movim in your language to make it available to everyone. You can join us on the Transifex platform (you can login using your Github account as well) and start translating directly from your browser.

Thanks to all the exisiting translators Movim is now available in close than 30 languages. Let's see if we can fully translate the project in even more than that! Some languages like the Dutch, the Russian or the Norwegian are only missing a few dozen sentences to reach the goal :)

Lets bring Movim to the maximum of people for 2018!

  • image
  • favorite

    9 Like

    odomar , florian.graetz , Timofey Kostenko , debacle , arie , ravenbird , JStargazer , Roelof Pieter , Alcides

  • 1 Comments

  • 28 February, 2018 arie

    Signed up for the Dutch translation team, let's see how I'll do ;)

  • Mo chevron_right

    Timothée Jaussoin – pubsub.movim.eu / Movim – Saturday, 20 January, 2018 - 10:01 edit

    A nice article, in German that present Movim, what do you think of the conclusion :) ? #hasecke #german #movim

  • link

    Movim

    Kick Ass Social Network on top of XMPP

  • favorite

    3 Like

    Paul Schaub , debacle , z428

  • 7 Comments

  • 20 January, 2018 Johannes Brakensiek

    thanks for sharing that blog post. I don't think that the number of users has anything to do with programming language either. it's rather about software quality, team communication and marketing. many successful distributed networks did have nice backing campaigns or were supported by successful American companies or people. many successful projects are American though. For a long time a problem of Movim was that it was quite focused on a French community, imho. But I think it's quite worth it to support a project that's mainly European and targets European privacy protection as well.

  • 21 January, 2018 Dominik George

    I renamed it to „Gemeinschaften“ for a reason ;)! For everyone knowing German TV commercials: Don't call it Gruppen!

  • 21 January, 2018 juh

    Thank you very much for sharing my post about Movim. There were some discussions on Diaspora about the post as well. And I hope that my cursory overview can attract a handful of new users to a project which is really amazing.

    The french centric community might be a reason, that I overlooked Movim until now. Only lately I recognized a couple of Open Source projects from France. The french open source community is a lot more active than I thought, but perhaps they are too busy with themselves. We need more networking in Europe.

    I don't think that calling Movim a European Project is a good idea. Mastodon is strong in China, USA and Europe. Internet and technology should not be labelled European. That narrows the focus too much. We need a European infrastructure of servers but the social network in itself is global. Why set boundaries if there are no?

    I cannot say a word about things Johannes mentioned: software quality and team communication. But being a marketing guy I would say that the website is a good start and the interface is looking nice. Perhaps it needs more drive. Some years ago Diaspora attracted new users as it presented itself as a Facebook alternative. Mastodon is about tweeting without Twitter. So people know what they want to do on the network. I must say that I really like Diaspora because there are many people with different interests. Diversity is great. Mastodon looks good, but it is quite boring. It's Twitter without all the users. ;-)

    Why does Movim exist and why should I care? These are the questions the network or the people who use it should answer. I don't have an answer. But looking at the foundation (XMPP), the website and Movim Hello Beta I would say that Movim could be the place where all communication come together: chatting, microblogging, telephoning, videoconferencing, live video news channels etc. This is a lot of things that need a rock solid, future proof, and scalable technology. This is why I am a little bit nickpicking about the programming language. Imaging an influencer with 1 million followers standing in front of the White House talking about the new hair color of the US-president. Could Movim bear that challenge?

  • 23 January, 2018 debacle

    Dominik, I'm German, but I don't know what is wrong with "Gruppen". Since long I don't have a TV set, so maybe I'm missing some context?

    juh, I had to laugh about "Mastodon: It's like Twitter without all the users"! I tried to use Mastodon and have account there, but I'm too stupid to understand the concept.

    My favourite social network at the moment is Diaspora, too, but I would be prefer a more solid technical base. A Diaspora clone based on XMPP would just be perfect :~)

  • 25 January, 2018 Johannes Brakensiek

    juh, thanks for your reply. Of course Movim should not be limited to European borders. It's just a feature it aims the needs of people in the EU especially well.
    Regarding the PHP technology I would suggest that the daemonized PHP architecture is somewhat limited regarding performance. (PHP itself is not - it is known to scale very well horizontally. But of course horizontal scaling is difficult when you use a PHP daemon.) But I think that's not the point: Having a dezentralized structure load can and should be distributed over a network being able to bear higher loads. So I think it's part of the ongoing development process to make XMPP and its pubsub servers be able to handle even higher amounts of people using it.

  • 29 January, 2018 juh

    Good point, Johannes. How does this pubsub federation works? I have the impression that content from a pubsub server is only fetched when I click on a link. The same what happens when I click on an external link on a website via HTTP. When I subscribe to a channel I would think that the content is preloaded when I lock in. That would speed up things.

  • person

    2 April, 2018 florian.graetz

    A nice wrap-up, although the last paragraph is a little discerning. However, the author pointed out correctly, that the key to social network is reaching a critical mass of users. The major reason, why new users sign up, is either that many of their friends are already members (that's how facebook drags in people), or that the new network provides outstanding features (that is how snapchat succeeded in the first place).

    So, why should new users favour movim over other decentralized social networks like Diaspora? The key features are basically the same. I hate to bug you with this again, but end-to-end encryption maybe an outstanding feature, which would even companies allow to use a social network for internal communication. This in turn could generate a steady cash flow. [And yes, I have read the issue https://github.com/movim/movim/issues/63, which discusses, why this is hard or nearly impossible to implement server-side.]

    By the way, all other decentralized social networks are leading similar discussions. In my humble opinion, the key is storing the private parts of the timeline on the user's device(s) instead of persisting them on the server. New messages would be pushed end-to-end-encrypted. Whenever new contacts are added, they would receive the entire timeline from the user's device immediately after the contact has been initiated. You have to keep in mind, that removed contacts may still keep your private data.

  • Mo chevron_right

    Movim 0.13 – Coggia – Christmas Release

    Timothée Jaussoin – pubsub.movim.eu / Movim – Sunday, 24 December, 2017 - 12:34

Two months after Movim 0.12 – Lovejoy here comes the new release of #Movim for #Christmas.

Rework of the Movim core engine, here comes ZeroMQ

This is maybe one of the biggest changes that is brought to our users for this version.

Since 2014 Movim is using a full real-time architecture which was slightly improved and optimized through the years. Movim Coggia comes with a totally new way of handling and distributing the events and messages in the core of the project.

ZeroMQ Logo

The daemon is now using ZeroMQ (and the related PHP library php-zmq) to take care of all those internal communications. This allows Movim to be fully multi-threaded and to properly prevent some user sessions to slowdown others if they are using the backend intensively.

Following those changes, many buffers were removed to unlock the full potential of #ZeroMQ. This means that the messages and events are now directly pushed to you (or to your server) without waiting lanes. Movim now feels way more snappier on all devices (especially mobiles).

Animations for a smooth UI

Related to those changes, some small #animations were introduced in the UI to make it smoother to load and be displayed. This allows Movim to push a little bit forward our goal: bring modern XMPP features in a light and fast interface to everyone :)

Here comes Ajax, again!

This version also brings back HTTP #Ajax requests in the project. Those will only be used in specific cases, mainly to ensure the good delivery of some requests to the backend.

The first place where this is now used is for the chat messages delivery, they were previously sent using Websockets. This previous method didn't guaranty the message was successfully handled by the backend. By sending those messages using Ajax requests we can now know if we had a network failure or some troubles on the line and then display a proper message to the user.

This small change is especially useful on mobile when the network is unstable and had proved to work quite well on 2G networks :)

Nigthmode, your eyes will love it!

This version is not only coming with changes in the project's pipes but also brings some nice improvements and features to our ever-growing user base.

The Movim Nightmode

A big refactoring was made in the color handling of the project, allowing Movim to now handle more easily custom colors (and maybe unlock "themes" in the future). The first step was then to offer our users a #Nightmode. You can enable it in your configuration panel :)

And because the configuration is shared between all the Movim instances you'll be able to have it everywhere you log-in.

Chatrooms

Several bugs were fixed in the chatroom session handling to take care of the different behaviors between the existing XMPP servers. A loading state was also added to notify the user to wait for the full connection of his account with the chatroom before doing anything with it (posting messages, inviting users…).

The connected users list is also now sorted by name and Movim gives you some proposal to autocomplete your chatroom address when you are typing it.

But also…

The login form has been slightly redesigned and simplified as well as the form to configure your Community (the advanced form is still available by pressing the menu button).

Redesigned login page

The Miho stickers pack has been fully completed and now includes 16 unique stickers for all our users :). The full pack has also been published on the LINE Creator Market, if you are also using LINE you can now support the project by buying (or offering) this pack of stickers.

The Likes are now collapsed in one little block below the articles to give space to the comments.

Collapsed likes to gives space to the comments

With the help and feedback of the users, many small bugs related to the articles publication were fixed.

ReactPHP, the library that Movim is using to handle the real-time part of the project has been updated to its latest version.

Help us to push Movim further in 2018!

In 2018 we would like to do everything to bring Movim to the maximum of users and show them all the possibilities offered by the platform. For that we will need your help.

You can naturally help us by directly contributing to the project by discussing new features and improvements on our official chatroom, fixing bugs or helping with the translations.

But we are especially looking for people that would like to talk about Movim online by writing articles telling how they are using the project, giving nice examples and tutorials on how to play with Movim and the XMPP environment and more generally spread the word about project.

Finally you can help us by giving a bit of money to cover our monthly costs for the servers, domains and other expenses. We hope that 2018 will see Movim and the Movim services to be fully funded by the community. To help us with that you can become a Movim Patreon or donate some money using Paypal.

We wish a Merry Christmas and a Happy New Year to all of you. See you in 2018 for more exciting features and surprises!

Happy New Year 2018

The Movim team.

  • favorite

    10 Like

    Matija Šuklje , Miguel A. Arévalo , Marzanna , JStargazer , d.link , invisible , paulfree14 , debacle , z428 , Baudouin Feildel

  • Mo chevron_right

    Movim 0.12.1

    Timothée Jaussoin – pubsub.movim.eu / Movim – Thursday, 16 November, 2017 - 22:26 edit

Movim 0.12.1 is a minor #release that follows Movim 0.12 – Lovejoy released a few weeks ago.

It mostly includes bug fixes and code #cleanup but also comes with a #redesign of the Community page.

The new Community page look

In parallel some work has already been done for the upcoming version, 0.13. All the internal inter-process communications (IPC) will soon be handled by the ZeroMQ library to fluidify the handling of thoses messages and make the UI more snappy.