Real-Time Collaborative State on the Edge

Hey, everybody. My name is Steven, and I've lived all over the place. So, I'm doing this talk from Paris today, but I lived in many different places. I spent, you know, 7 ½ years in Australia, in Sydney, even lived in Quebec before that for a little bit, and also spent a few years in New York City.

So, yeah, great to be here today. And what I want to tell you guys is that I am a massive design tool nerd. I love building. I love designing design tools. I love creating apps and things that people use as a return and create something with that tool that I helped make. And so, this is the angle for my talk. I'm a big design tool nerd and I love making this kind of creative tools.

So, I worked at Facebook Messenger for a little while on the design system slash Messenger team to help them with their tooling needs internally. I also spent some time at a company called Campaign Monitor in Australia back when I was in Sydney. And, my main project there was to work on this email builder. So, it was a drag-and-drop tool where you could design beautiful emails that were like responsive, so they worked on mobile and desktop. And we helped brands kind of create those emails just in a drag-and-drop way very easily. That was the first time really that I was really introduced to creative tools. That eventually led me to create Easy. And Easy, I don't know if some of you have heard of it, it's pretty old now, was a tool to help designers create animations for the web, and then to do that without having to code. And then you just could click that and get the CSS from it. It was pretty cool. I enjoyed working on that. And that eventually led me to join InVision.

And so at InVision I worked on the team responsible for creating InVision Studio. I worked on that for four and a half years. And I don't know if you guys are familiar with InVision Studio. But that was essentially a UI design tool where you could, you know, obviously design apps, websites, but also create animations for them and prototype them, make them very interactive. And this sounds pretty obvious, but back in the day there weren't very many tools that could do that sort of stuff. And really, that experience really led me to the path I'm on today. We had a, you know, it was a desktop application working on the file system, and Figma was this amazing browser based tool to do UI design and that sort of stuff was starting to become really popular.

And one thing that Figma did really well is that, as I said, the product worked in the browser, meaning you don't have to download an app to get it to work, and it was what they called multiplayer. And so, we eventually had this project at InVision to convert InVision Studio from a desktop based file based application, excuse me, to an application that was browser based and collaborative by default. And so I led design for the project, and this is the first time I really got exposed to the problem of like, damn, this is freaking hard to do. We had a dedicated squad of, I think, at the peak, seven engineers, one point manager, and I was the designer of kind of heading up that initiative. And it took over it took about a year and a half before we could have a working version of that desktop based, sorry, browser based multiplayer application a year and a half. That's a very long time. And, yeah, I was eventually a little bit frustrated with this because I was like this, I just don't understand how this could take that long. But that is that's just how it is. Like it takes very long to build those kind of experiences when you do all that stuff yourself. And so being exposed to that I didn't realize, but that eventually led me to the path I'm on today. I run a company called LiveBlocks and at LiveBlocks what we do, we provide APIs and tools to developers like you guys to build multiplayer experiences, but also more broadly collaborative experiences like comments and pretty much anything you want to do that's collaborative and requires multiple people to work together on some type of digital experience. But don't don't be worried. This is not a sales pitch. I promise I'm only here to share my learnings and dive into the behind the scenes of those kind of infrastructures, the kind of infrastructure needed to build those kind of experiences. But yeah, just to show that this problem that I personally faced with my co-founder Guillaume at InVision has become I mean, it's always been an issue, but most recent years there's been a lot of technologies and frameworks to do to work on this problem. And so a lot of those frameworks and tools when they talk about this often refer to multiplayer. But in the world of collaboration, when you build a collaborative SAS product, multiplayer is only the tip of the iceberg. There are a lot more things to think about when building a collaborative product, right? You need to think about, okay, well, when my users are in my product, how can they leave comments? How can they mention their co-workers or collaborators in comments, in specific annotations within the document, so that those people end up collaborating asynchronously and can join the document and be notified elsewhere? There's also shared dialogues, like how do you share an artifact or something with somebody else? How do you add those people? How do you take care of the permissions for those people? And then, obviously, once you have those documents and those collaborative artifacts working, how do you browse them? How do you find them? How do you find all those documents? How do you share those set of documents? How do you put them into a folder and that kind of stuff? So document browsing is a big part of that. And then, obviously, things like audio and video are becoming more and more present. So if you want to do an audio huddle, you can do that sort of stuff. Share like a video as you're working on something. Having that experience directly in the tool is becoming more and more relevant for a collaborative experience. And so if we take a step back and think about how we got to the state of where we are today in terms of collaborative SaaS products. To me, it really started with Google Docs. I think that was 15 or 20 years ago at this point. They introduced this is not the screenshot. You see, this is a lot more recent, but they introduced Google Docs, a browser-based document or similar to Microsoft Word. And it was mind blowing. Like you could just open a URL and that URL was a Word document pretty much. And you could edit it.

You could copy that URL, paste it to somebody else. Pasted it into somebody else, and that other person would be in the same document with you collaborating. And then probably a decade later, Figma started working on their product. They were creating this design tool, made it browser-based and what they did with Figma is that they revolutionized the design industry so that people could work directly in the browser. And now not only designers could actually design files or have access to those design files, they made the design process available to a lot more than designers.

Now you have developers coming into Figma to inspect the elements in order to help them build those UIs that designers put together. You have stakeholders, founders, you know, C-level people that can go into files, review and experiment with design. And you even have, like, product people or pretty much anybody that wish to do so can just go into a file and play around with some of the components that are pre-built by the design team and start actually designing some of that stuff themselves. Figma has been really, really successful. And I think what they showed with their product is you can actually make a high quality, performant tool that typically was only desktop based. You can actually put that in the browser and make it collaborative. And all of the sudden you get so many benefits organization to seamlessly collaborate together in one space. And there are obviously more companies nowadays, you know, that have those type of multiplayer kind of collaborative experiences by default.

Canva is one of them, Mural, obviously for whiteboarding, Notion, Mirror, Linear has been really great at that for the ticket management software and obviously Pitch to do presentations like this one. And to enable multiplayer collaboration, you need to think about conflict resolution. And what is conflict resolution? Essentially, when you are in this world where multiple people can manipulate the same document state, you need to be able to resolve. Conflicting actions from different people that are being processed at the same time. And so if you have somebody requesting to make that rectangle red and somebody requesting to make it a circle, like what do you do at that point, you know, do you make it just red or do you combine them make it red and make it a circle. You need to think about those things when you're in the multiplayer world.

And to do that, there are two main algorithms really to enable this. So operational transforms, which modify the operations is typically used by Google Docs to use OTs. And then you have CRDTs, which stands for, I never remember, Conflict Free Replicated Data Type, and those modify the state of the document. OK, well, that's pretty confusing and I'm very much confused, too, it's a lot of acronyms and it's pretty hard to use. And so there are some CRDT frameworks out there that do this really, really well. Yjs is one of them, and they essentially provide you with different data types like right array, right map, right text and more to kind of take care of that complexity and make sure that if you interact with those data types, you don't have to worry about conflict management. Yjs will kind of handle that for you. Auto merge also is a one that's great, comes with different data types, counters, lists, text values, and they basically handle all that stuff for you as well. And they're pretty easy to use, right? Like, this is just an example of why just if you want to do an array, you just do a new wide array, push the content to it and shift the content, and you can also delete an index. So it's very similar to how you would interact with a standard JavaScript array, except that they make sure that everybody always going to be seeing the same data, even if there are multiple people manipulating that array at the same time. The thing, though, is that CRDTs can be hard to use and scale in production.

You have to think about data migration, like, you know, if you're a serious production application, you want to be able to say, like, my data is going to look like this and you want to be able to evolve that over time. So that can be pretty hard to do. Schema validation, you know, very similar to data migration that kind of linked with one another, but being able to update the schema of a document over time as you're as your product evolves, it's pretty pretty important as well.

And so one of the big things as well with CRDTs is that the data can be very, very complex because most of the like all the changes typically get stored. And so instead of having one JSON representation of of a document, typically you have all the changes that happen over time. And that often leads to a potential, very large document size, even if the visually like the actual JSON representation can be small, that JSON representation is made of all the changes that have happened. And that can lead to like a pretty big file, especially if that file has been around for years and many people interacted with it.

And so what we just looked at is just the tip of the iceberg. It's only the multiplayer part. There are a lot more things to think about, as I said earlier. And so if you look at I just selected like four top kind of SaaS products that do collaboration really well, they all have similar patterns, right? If you look at Google Notion, Figma and Pitch, they all have some kind of shared dialogue where you can invite other people, copy the link, and also set specific permissions for those people. Like, can this person edit the file? Is that person the owner of a file? Can they only view that file? Those are the type of things that that multiplayer products need to enable as well. Comments and mentions. Like great collaboration needs comments.".

This is just a small sample, but that list goes on and on and on and on. And you can scroll for probably a couple of minutes without reaching the end of those changes. And if you look at kind of how those changes get assembled, they look very much like this.

This is a tool that we built. It's a DevTools browser extension to enable you to inspect your Yjs experience to understand how the document has been constructed. You can see kind of the dependencies between the changes and what leads to that representation of the data. And so, yeah, pretty simple data at the end. But the actual representation itself is pretty simple. But the actual data making that representation can be can be pretty complex and lead to pretty big, pretty big file size in the end.

And so what we just looked at is just the tip of the iceberg. It's only the multiplayer part. There are a lot more things to think about, as I said earlier. And so if you look at I just selected like four top kind of SaaS products that do collaboration really well, they all have similar patterns, right? If you look at Google Notion, Figma and Pitch, they all have some kind of some kind of shared dialogue where you can invite other people, copy the link, and also set specific permissions for those people. Like, can this person edit the file? Is that person the owner of a file? Can they only view that file? Those are the type of things that that multiplayer products need to enable as well.

Comments and mentions. Like great collaboration needs comments. Like if you want to be able to collaborate, chances are you're not always all going to be in the same file at the same time. You want to be able to comment and collaborate asynchronously as well. So Google, you can comment, you can mention people and that's the same across the board. And then document browsing. All of them have some type of workspace view, you know, where you can browse your documents together with other people. You can create folders and those people can access those those folders and browse the documents. And then not all of them, but I think this is a pattern that we're going to see emerge in more and more collaborative tools. Audio and video plays a big part into some of the tools like pitch, for instance, enables you to have video right in your presentation so that as you move around your cursor, your face is essentially attached to the cursor, giving all the people in the document a sense of presence and a sense of being together in the same room with you. And then if you look at Figma, Figma has this audio huddle feature, enabling you to quickly get into an audio call with somebody else that's looking at the same document as you. Making it very easy and seamless to collaborate with those people in real time. And so if I look at all those companies, they all do a great job. But I think there is one that definitely stands out for me and has been super inspiring for what we do at Liveblocks. It's definitely Figma. I think Figma has been great at assembling all the required collaboration features in a very nice way so that they seamlessly work together.

And as a user, you end up having a great collaborative experience, almost as if you were in the same physical space with somebody else. And so if there is one blog post that I would recommend from them and especially from Evan Wallace, who was the co-founder of Figma and the CTO for most of the for multiple years there, he explains how the multiplayer technology works. And it's a brilliant blog post with like a lot of interactive illustrations that makes it very easy to understand how that stuff works.

And so the work that Evan and Figma did eventually led Guillaume, my co-founder and myself to this question. What would developers build if they could access Figma's infrastructure through React hooks and components? This is something that I really wish we had back at Invision when we were converting that desktop based application to a browser based multiplayer collaborative application. And so eventually we were like, damn like that would be... This is so hard to build. Like, why don't we why don't we have that as React, some React hooks and components and make it very easy for like front end developers to build those kinds of experiences without having to worry about all that complex back end infrastructure? And so that's what that's what it is to create live blocks. We kind of stumbled upon this idea as we were working on a video presentation tool. And as we worked on that tool, we realized, damn, this feels very much like what we experienced at Invision, like we're spending awful time trying to figure out the real time aspect of things and very little time actually focusing on the tool. And so at that point, that that's when we decided to create live blocks.

And so once again, I promise this is not a self-speech. I just I just want to show you a bit more. The practical side of this, like to show you some of those kind of multiple experiences, how the infrastructure works behind the scenes and how we enable that at live blocks. And if you think about the collaborative experiences that I highlighted before, they all work in the same way. There is one central artifact on which multiple people collaborate and that artifact can be a text document. It can be a whiteboard. It can be a creative tool. It can be pretty much anything you want. And to build that central artifact, there are multiple things that you need to make it collaborative. You need a concept of presence so that the people looking at that artifact know who is currently currently looking at this document, showing the live cursors, like showing the even eventually like showing the video of the person next to the cursor like pitch does, showing like a live avatar stack, showing what item people have selected in the document, that kind of stuff. Then you need the concept of broadcast. Right. So when you are in a room, you want to be able to broadcast events to the other people that are currently in the same room as you. So if you want to do things like live reactions, for instance, a fire emoji on this element, whatever you click it, you broadcast an event and other people can then show that fire emoji at the same time. And then this is the core of the document itself. You need to you need a data store to store the document of the data and that data store needs to enable people to make edits to the same data at the same time. And that's what CRDTs enable like Yjs or like auto merge or some other frameworks out there. And then lastly, you need comments. So comments is more for async collaboration, but you want to be able to comment and annotate on specific parts of that document so that when you mention somebody there, that other person receives a notification and they're incentivized to then go into the product and collaborate and see what the other people have done.

But this is a pretty short talk, so I will only focus on the document part and how that works. So at LiveBlocks, we have two ways to store documents, so we have our own proprietary solution called LiveBlocks Storage, and we also recently introduced a way to store Yjs documents right into the LiveBlocks infrastructure. And so if we look at storage, we have three data types for now. So LiveList, LiveMap and LiveObject. And I wanted to dive into the list use case because I think this is one that's nice to highlight visually.

So if you take a list, there's three elements in that list, A, B, C and you have one person that on the left moved the B list item to the last position and the person on the right decided to insert a new item in that list between B and C. Now there's a central server here. Like what is the truth of this document? What is the new order of that list knowing that two people made changes to that list? Well, it's hard to figure that out and the solution is fractional indexing. And the way this works for instance here is that each item has a fractional index and that index is used to define the order of that list. And so if somebody on the left decides to put the second element at the end of the list, that new element is going to get a fractional increase so it becomes 0.4. On the right though, this person decided to insert that element between the second and the last item in the list. And what this does is that it takes the previous item and the next item and finds the middle. And so that's why it becomes 0.25 and then the server can then now reorder that stuff and find the right order. So 0.25 is the second item and the pink one, 0.4, is the last one and that's the reconciliation we see. And boom, the two people then get exactly the same list once it's been resolved. And this is pretty easy to do with LiveBlocks storage. So you can create a new LiveList. It's very similar to like a standard JavaScript list except it's a LiveList from LiveBlocks. And here you have an order. You can then decide to insert an item into the list and convert that to an array and this is the result you will see. So this, in this example, looks exactly like a normal kind of JavaScript array. But here you can imagine multiple people could be doing this at the same time and we take care of the fractional indexing behind the scenes to make sure everybody sees one unique list ordered based on like all the actions that all the people have taken on that list.

But then there's another obviously big part of this, which is pretty, pretty complex and that's something we spend a lot of time on is the infrastructure powering the real time side of things and making sure that all that stuff scales very nicely. And if you've worked on WebSocket servers before, you would know that maintaining them and scaling them in production is very difficult. They're really hard to scale, hard to manage, managing the connection can be can be challenging and there are a lot of edge cases.

And so this is just an example of our, this is a simplified state machine that we put together with Stately, which I highly recommend. It's a fantastic tool. And this is the state machine we use, simplified again, to handle WebSocket connections at Liveblocks. And there are a ton of edge cases to think about. For instance, you know, when you finish your day at work, you're going to close your laptop and then in the morning you open it back up again. You actually need to do to restart the WebSocket connection, you need to do a ping pong mechanism to check if the connection is still active. There are a lot of edge cases like this that were pretty hard to map out. And we used this state machine to create kind of all the edge cases and make sure that as a user using LiveBlocks, you don't have to worry about thinking about the WebSocket connection, we handle all that stuff.

And in order to scale this, you know, and not have a single point of failure, each collaborative document, we call them rooms, they live milliseconds away from people on the edge. And what's great about this is that it gives people an absurd performance, like, you know, in most cases you're 10 to 60, 80 milliseconds away from the server, meaning you can get really fast real-time performance. But not only that, it enables us to scale this horizontally, essentially, because there's not one single point of failure and we can have as many rooms as we want. It's just a central server on which people collaborate.

And that was it. I hope this talk was enjoyable. I hope you learned something, and yeah, thank you so much for listening. Cheers. Bye-bye. Bye-bye.