Matrix Live S08E17
|Faster Room Joins… Again?|
|Date of Upload||27.01.2023|
Topics of the Episode
- Progress on Faster Room Joins
Original Description of the Episode
Today my guest is Mathieu from the backend team, to talk about faster room joins. Again, you may ask as an avid follower of Matrix Live. Again! Let's talk about Bugfixes and Improvements™
🪙 Donorbox for individuals - https://donorbox.org/keep-matrix-exci...
💶 Matrix Foundation (paid) membership for orgs - https://forms.gle/Yy345QkB5pifJJNy6
(This Week in) Matrix
Thib: Hello and welcome to Matrix Live, the first episode of Matrix Live that is actually live in weeks, if not months, if not years. They have been trying to take OpenTurtle savers away from me, but I will be live with you.
So today my guest is Mathieu from the backend team. Mathieu, bonjour. Mathieu has been working on faster room joins. And Mathieu, room joins are one of the most painful performance issues of Matrix when it comes to being the first person on your home server to join a room, which is actually something that doesn't happen a lot on matrix.org. So this is mostly going to be a fix that is going to be a life changer for selfhoststers like me. We already had an episode with Rich on fast room joins when the effort was kickstarted, and it seemed to be well on its way.
Can you remind us what Fast Room Join consists of from the user perspective? So let's not dive in the details right away. And what user experience problem does it solve?
Mathieu: So yeah, like as you said, on matrix.org it's not really a problem because the server has already seen a lot of the rooms already. So at least the big ones, the big public ones, which is the main problem here. But if you are on a small server, and you are the first user to join, it can be really slow, especially in a big room like HQ, so you have some kind of spinner of the death.
It usually times out.
You refresh, it's still not there.
You're hanging there.
You don't really know you're in the void.
Did I join?
Did I join?
I don't know.
And then later on you get the room appearing in your room list success, but it was not really a great experience.
So you mentioned HQ, which is the family nickname for Matrix HQ, which is one of the biggest matrix rooms we know, like 40,000 users or something like that.
So is it just the number of users that makes it difficult to join, or is it because of something else?
Mathieu: It's mainly the number of users, to be honest. There is something called the state of the room, which includes if the user has joined or not, if it was invited, for example, or even a lot of other potential information. But the bunch of it is really what we call the membership events, which is as a user being joined, kicked, or stuff like that.
So on the previous episode on Fast Room Joins, it was a while ago, I think it was in late 2022 with Rich. It seemed to be well on its way, and it can't really be that difficult because what you are trying to do is try to find out who's in the room, maybe not display the latest members and display a handful of last messages. So it's not that difficult. Why wasn't that sort of just in an afternoon?
Mathieu: Yeah. So the program is not really displaying the last messages, for example, that's pretty easy. But we live in a decentralized world in Metrix, and it's complicated to handle. You need a lot more info than just the last messages or the couple of last joiners or the people who just talked in the room as a first info. At least as a first info to be able to send something in the room. Reading is not really the trouble, it's more sending. Because you need to be able to, your server needs to be able to tell if you have the right to be able to send in the room, either because of power levels that could say this room is read-only for everybody except a couple of people, or if you have just been kicked, for example, or banned from the room. Banned, sorry, not kicked. So yeah, and these rules need to be - to apply these rules, you need to fetch a lot of events. I mean a lot. A bunch of events, not a lot.
But by default, we say, when joining a room, we get a full list of people. And this is, however, unneeded because we don't need to know that everybody that is in the room knows if you can talk or not there. So it doesn't look that complicated, either. But then you have a lot of fetch cases in various components of Synapse itself, or even matrix protocol, which makes it a bit tedious.
Just to give you numbers, for Matrix HQ, the global state is 200 megabytes of uncompressed region. So it can seem not that big, but in the end, you still have to collect all of the data in the DB, and then your homeserver has to ingest it and checking the rules on the way when ingesting all those events, and that can be quite long.
[Unclear speaking over each other]
I was going to say that you have a lot more info in the blog post that Rich did in October, or in the MSC 3902, which was also written by Rich, which goes into a lot more detail if you want.
So I remember you mentioning to me one issue in particular that was about banning users or not having access to a specific room. It could be easy to do things carelessly and just join users to a room really quickly. But if you don't do things properly, you risk leaking information, for example, to people who were banned from a room. So what you are telling me actually is maybe it's not technically that difficult, but there is a lot of edge cases that need to be considered in order not to erode the trust that people put in Matrix.
So this is an exciting change and me, as a self-hoster in particular, I'm really excited to try that. I must confess, I'm not in Matrix HQ because I have a single person instance, and it's been terrible to try to join that room. When can I try that? And what do I need to do to enable it?
Mathieu: So it should be live with all the last fixes and tests that we did recently on 1.76 Version. We plan to enable it by default there. Once you update to that, you should have it by default.
Thib: So 1.76 is the next version that is going to be released early next week?
Mathieu: Yes, it is.
And so you already have a Boolean that you can put to true. We plan to default that to true for the next version. You can still go back and put that to false quite easily, in case of. So in case there is something wrong, it's quite easy to disable it. But it should be mostly safe in the sense that worst case scenario, you will not be able to send messages during the first join, which is still a better experience, I guess, and nothing will work and not even seeing the room. So yeah.
Thib: All right.
So it's going to be enabled by default. And if I want, I can opt out of that. This is going to be in the release notes, I believe, the setting to set to false. So this is something I can use on my production server, even if I'm, I don't know, a large entity that relies on matrix and really, really, really needs matrix to work. You are feeling really confident that this is going to work. This is not a hack.
Mathieu: I don't think it's a hack, no. It's complicated, but it's not a hack. Not much else to say. I didn't design the stuff. I'm new to the backend team, but some other people are way smarter than me did it. And I have confidence that it will be fine.
Is there anything else you want to add to conclude the episode, which was quite short, but anything you want to hype us about?
Mathieu: Oh, yeah, right.
I have a talk about that at FOSDEM. So it will be in the matrix track. I don't know exactly the name, but...
Thib: It's matrix-devroom, yeah.
Mathieu: Yeah, matrix-devroom. It's planned to be recorded because we only have half a day of live ones, but it should be available on the internet pretty soon. And I will go a lot more in technical detail there. I will go through some edge cases, for example, that I didn't really have time to go here.
Thib: Okay, fantastic. Thank you very much, Mathieu.
To conclude the episode, I would like to give you, the community, an update on our call to support us financially. So individuals are answering the call and keep answering the call. I get regular updates in my mailbox that people are making donations to our donor box.
Thank you very much.
It means a lot to have a community that's behind us, that believes in our mission, and is ready to support us.
We have a few organizations who showed up to support as best as they can, mostly small to medium businesses. They are able to donate a few hundreds to thousands of euros, which is a significant commitment for them, given their size. And we really appreciate the efforts.
Thank you very much as well.
There is one organization that showed up to make the kind of commitment we'd expect all large organizations to make to allow us to go full steam on Matrix. We're still in talks with them right now. But other than that, large organizations, sorry, did not really answer the call yet. We expect to have to solicit them individually and share a more personalized message to them. This is time we'd love to spend working on improving Matrix instead, but we will get to it.
If you work for an organization that relies on Matrix, you can help us, please advocate internally for a membership to the Matrix Foundation. We need it to keep the Matrix ecosystem in good shape, to work on performance updates, and more generally to make Matrix a joy to work with.
As usual, I'm leaving two links in the description, one to the donor books for individuals and one to the membership form for organizations.
In the meantime, take care of yourselves, and I'll see you next week in Brussels at FOSDEM.
- ↑ van der Hoff, Richard (3 October 2022). "MSC 3902". Retrieved 12 February 2023.
- ↑ "Synapse 1.76.0 (2023-01-31)". 31 January 2023. Retrieved 12 February 2023.