Matrix Tutorial 7

From Miki - Matrix Wiki
Jump to navigation Jump to search
Matrix Tutorial #7
Getting Ready for Sliding Sync
HostThib
Date of Upload27.01.2023
Video Platforms
Youtubehttps://www.youtube.com/watch?v=25wkV2ZCSsM

Shownotes

Topic of the Episode

Useful links

Original Description of the Episode

You can't wait for Sliding Sync and Element X to be released? We can't either. The public TestFlight of Element X is going to be announced in https://matrix.to/#/#element-x-ios:matrix.org In the meantime, get ready by deploying your sliding sync proxy!

Support Matrix!

🪙 Donorbox for individuals - https://donorbox.org/keep-matrix-exci...

💶 Matrix Foundation (paid) membership for orgs - https://forms.gle/Yy345QkB5pifJJNy6


(This Week in) Matrix

🐘 https://mastodon.matrix.org/@matrix

🐦️ https://twitter.com/matrixdotorg

[m] https://matrix.to/#/#thisweekinmatrix:matrix.org

Transcription

Hello and welcome to this new episode of Matrix Tutorials. Today we're going to deploy a sliding sync proxy.

If you have an account on Matrix.org this is not necessary for you. The proxy is only useful for people who host their own home server. Matrix.org already has one and it works out of the box.

Sliding sync is an upcoming change to the matrix specification you might have heard about at FOSDEM during Matthew's Matrix 2.0 talk. If you haven't yet I strongly recommend watching it to know everything we've been doing.

If you want to dig deeper in sliding sync and how it works I also recommend the FOSDEM talk "Clients as Good as You'd Expect" by Kegan, Ben and Mauro who cover it end-to-end from the spec and server side to the SDK and client side.

For the sake of simplicity we are going to build on top of our first ever Matrix tutorial, the Docker Compose one. The concept will remain the same whatever your deployment method is.

A graph showing a reverse proxy accessed from the outside which then distributes the request either to synapse or an nginx. From synapse there is another arrow going to a PostgreSQL. Aside on the right there are another 3 Boxes saying "Synapse Data", "PostgreSQL Data" and "Nginx Data" from top to bottom.
The base setup used in this Tutorial

So this is what we have deployed during the first tutorial. [Editors note: See image to the side] The reverse proxy synapse and its Postgres database, a Nginx to serve the .well-known file for the delegation of incoming traffic. This was all hosted on a single home server and when your client connects it connects directly to your server.

The sliding sync proxy we're going to deploy is a proxy so it sits in front of your home server. With that said the sliding sync proxy can be hosted on the same machine as your home server.

So you might wonder why there is a sliding sync proxy in the first place if we're going to deploy it on the same server. After all it's supposed to be the next step in the spec. Well it's because of the next step part. Sliding sync is not finished yet and still needs to be worked on. It's much easier to work on an independent piece of software rather than to have to maintain a branch of Synapse or Dendrite or Conduit.

The proxy acts as a sort of buffer client. Without anyone connected it doesn't do anything. The first time you connect to the proxy you give it your access token. The sliding sync proxy then uses this access token to do regular sync against the server and gets a local copy of your account as would any other client.

This is why the first time you connect using sliding sync will be probably slower. But then the sliding sync proxy will stay in sync with your home server. Next time you open your client it will already be up to date and serve you the most important content first. Your UI will update much more quickly than with a regular sync.

Eventually sliding sync is going to be merged in the spec. It will make sense for Synapse, Dendrite and Conduit and other home server implementations to support it and the proxy will become irrelevant.

The image is mostly the same as the previous slide. Added with a blue dashed boarder are 3 cards. The first being the "Sliding sync proxy" which has a subtitle saying "matrix.example.org". Next to it are a "Proxy PostgreSQL" and a "Proxy PG Data" card.
The changes made to the setup in this Tutorial

So this was our deployment. Today we're going to add a sliding sync proxy, its database and a volume for the database. [Editors note: See Image to the side]




A typical well-known file consisting of the base_url
.well-known content before it was updated

The sliding sync proxy itself doesn't store anything outside of its database so it doesn't need a volume. We're also going to update the well-known file served by the Nginx. This is the file we used to serve to tell people that our Synapse was actually here and this is the location of incoming traffic. [Editors note: See Image to the side] If you don't remember it you may want to watch the tutorial number 1 again.


A well-known file including both the base_url and a url field inside of a "org.matrix.msc3575.proxy" object
.well-known content after it was updated

And this is what we're going to add. [Editors note: See Image to the side] We're going to tell clients that we have a sliding sync proxy and this is where you can find it. So let's get started.

Alright so what we need to do first is to open a browser and a search engine and look up matrix sliding sync and we're going to run on matrix-org/sliding-sync so the repository of the sliding sync proxy.

We have some install instructions and a lot of things in the readme but what we are going to look for are the releases on the right side. So we go to the releases and it says Docker images available at this address which we can click on and we have an image that we can use in our Docker compose.

So I opened a terminal on my server and I have my Docker compose at the root of my root user directory so please don't do that. It's obviously not production ready this is just for the sake of demo

I'm opening this Docker compose file so this is the same one that we ended up at the end of the first matrix tutorial. If I go to the bottom of it I'm going to paste a few lines so we are going to examine that together. 

ssync-procy:
  image: ghcr.io/matrix-org/sliding-sync:v0.99.1
  restart: unless-stopped
  environment:
    - "SYNCV3_SECRET=asuperlongrandomstring"
    - "SYNCV3_SERVER=https://matrix.chipchop.org"
    - "SYNCV3_DB=user=syncv3 dbname=syncv3 sslmode=disable host=ssync-db password=asuperlongunguessablepassword"
    - "SYNCV3_BINDADDR=0.0.0.0:8888"
  depends_on:
    - ssync-db
  labels:
    - "traefik.enable=true"
    - "trafik.http.routers.ssync.entrypoints=websecure"
    - "trafik.http.routers.ssync.rule=Host(`sliding.matrix.chipchop.org`)"
    - "trafik.http.routers.ssync.tls=true"
    - "trafik.http.routers.ssync.tls.certresolver=letls"
ssync-db:
  image: docker.io/postgres:15-alpine
  restart: unless-stopped
  environment:
    - POSTGRES_USER=syncv3
    - POSTGRES_PASSWORD=asuperlongunguessablepassword
    - POSTGRES_DB=syncv3
  volumes:
    - ssync_db_data:/var/lib/postgresql/data

What do we have? So I declared another container the sliding-sync proxy. The image is going to use is actually the link we have here well the link the instruction we have here. So the main difference we have is I use v0.99.1 instead of main because I usually like to do the updates myself but it's really up to you and how you manage your containers.


So restart unless stopped and then we have three environment variables. The first one is a single three secrets so it's a super long random string obviously use something better than that in prediction. That is going to be used by the sliding-sync proxy. It's not the same as the postgres password.

Then we have the syncv3 server so this is the address where your synapse or dendrite or whatever homeserver lives and this needs to be the same as you have in your well-known file. So for example if I go in my case https://chipchop.org/.well-known/matrix/client. I have the base URL so it needs to be the same here in the syncv3 server.

Then I have the syncv3db this is the connection string that the sliding-sync proxy is going to use to connect to the database. So I decided to create a user called syncv3 then I created a database called syncv3 as well. The host is going to be the name of my docker container of the postgres database is going to connect to and then the password is the password I'm going to use.

So fantastic I have that set in place. I need then to add the right traefik labels to expose that sliding-sync proxy on the internet. So of course you need to make sure that you have a DNS entry that matches so an A record. I decided to use sliding.matrix.chipchop.org because it makes sense for me but you can use whatever you want.

Then I have a simple postgres database. So it's a postgres-15. Environment variables I have the user "syncv3" this is the user I'm using in the connection string above. The postgres password is the one I use in the connection string above as well and same for the database name and that container needs to have a volume so I'm going to add it at the bottom as well. The volume is a "sync_db_data".

A log with "panic: dial tcp <ip>:<port>: connect: connection refused" at the top
Screenshot of the log of the sliding sync proxy

Right and I can give it a go and do a docker-compose up -d. It's going to start everything. I can do a docker ps and my sliding-sync proxy is restarting so what is happening. We could look docker logs root-ssync-proxy-1. [Editors note: See image to the right for log output]

I know oh yeah so this is something that can happen so it has restarted because the way we specified it is not really production ready I just said that the sliding sync proxy needs well depends on the postgres database but as such it does not wait for the postgres database to be ready so it can crash at first but because I asked it to restart regularly the sliding sync proxy is restarting and at that moment the database is ready and the sliding sync can connect.

So that's cool we can have a look at the logs docker logs -f root-ssync-proxy-1 and so we can see that the sliding sync proxy managed to connect with the database it's listening to everything on the port 8888 and it does nothing as such. Right so what we need to do then is to update the well-known file so for that we are going to go to the root of the sliding sync git repistory we can go to the docs folder and there is a landing.md file which is going to give us the value that we need to add to the well-known file. So obviously you need to upgrade to change the URL here to match what you need.

So let's have a look I need to go to the Docker volumes in my case I've got the root nginx.conf if you followed the matrix tutorial number one it's not going to be root nginx.conf for you it's going to be infra nginx.conf but the rest is fairly similar so I can update the default.conf file and under /.well-known/matrix/client - not server - I can add what I was given on the page. So in my case I already changed it to sliding.matrix.chipchop.org because this is what makes sense for me then I can do a docker restart root-nginx-1 and I can check that it actually happened and I can see the URL here. And something very interesting happens if you do that if you click the URL actually you can see bad gateway so what does it mean let's have a look at the terminal. So we can monitor what is happening if we do a docker logs -f root-ssync-proxy-1 so we can see that if I have my logs here and if I try to reach the page nothing happens in the log. So what happens is that the reverse proxy is not able to communicate with the sliding sync proxy at the moment. So traffic does not make the connection up to the reverse proxy.

So what we need to do is very simple we need to just tell traffic that the sliding sync proxy is actually listening to the port 8888 so for that we just add a service load balancer server port we just specify the port.

- "traefik.http.services.ssync.loadbalancer.server.port=8888"

I can then do a docker-compose up -d then I can pull the logs of root-ssync-proxy-1 and now if I get back to my browser and reload I don't have a bad gateway I have a 404 and if I look at the logs as I refresh the page I can see that the sliding sync proxy is actually reached so that's very good this is exactly what we wanted.

We have deployed a sliding sync proxy and you can say oh but how do I know it works. Well you know it works because you have Element X installed on your phone. So let me record what I'm doing right now. I'm opening Element X and I'm clicking on continue and if I go to chipchop.org and press next it's going to ask me for my details and so I'm going to enter them. I think my password was that.

So yeah just a note here that your your home server needs to have registrations it needs to have - how to say that - not just SSO you need to have password-based authentication supported otherwise element X is going to fail.

A screenshot of the sliding sync proxy working

So I display the logs here so you can see what's happening you can see that the sliding sync proxy has started pulling my homeserver. So Element X has acquired an access token by doing a password-based authentication against my server then it's handed that access token to the sliding sync proxy and the sliding sync proxy is buffering and now what you can see on my phone screen is that nothing is being displayed which is not super impressive for sliding sync but as I told you in the introduction this is perfectly normal this is just because for the first connection the sliding sync proxy needs to buffer.

So we are going to give it some time to buffer and then it's going to make it.

So I hope you enjoyed this tutorial I hope you're going to make it to good use and that you are going to test Element X and give us a lot of feedback it's a huge huge improvement in matrix on the performance side I hope you enjoyed that please bear in mind that a lot of this work on the sliding sync part was done by the matrix.org Foundation so if you can afford to support of work please please please support us you can go to the matrix.org Homepage and go to the supporters section and give us a few euros, pounds or dollars if you are an individual or you can become a member of the matrix.org Foundation if you are an organization yourself and that you benefit from matrix I hope you enjoyed the tutorial and I'll see you next week see you around matrix live!

Retrieved from "http:///w/index.php?title=Matrix_Tutorial_7&oldid=5812"