Hacking into Labeled Arrows

Before I actually start, let me just give a quick intro. So I'm Akansha, I'm an open source maintainer, FOSS basically stands for free and open source software. I'm a core maintainer of Excalidraw, ReactJS, and a couple of more open source libraries. And I also do these FOSS meetups in Bangalore, India to promote the FOSS community in India. And yeah, these are my socials. I would love to connect with all of you and chat about anything frontend, React, open source, Canvas, anything.

So yeah, now let's get started with the talk. So what exactly do I mean by Labeled Arrows? Labeled Arrows are basically when you have an arrow and you have some text attached to it, that is basically a Labeled Arrow. You would have used it in some conversation as you can see the conversation between two friends or some communication, some flow, some architectural diagram to show some directions, some crucial information. It can be anything, but it is like one of the most used features. So since we are going to implement it, so let's look at how this Labeled Arrow is constructed internally.

Now this is a Labeled Arrow. It has a line. It consists of a line. It consists of an arrowhead and a text. So at least when constructing a Labeled Arrow, you have at least three separate different types of elements. And let's make a note of it because we are going to deep dive into this.

Now, as we developers do, like before building any feature, let's just come up with some minimal requirements to build a prototype. So let's look at the list of requirements. So the first is we want to understand how we can render the arrows in Canvas, how we can render the label that is the text on top of the arrow, how we can link the label and the arrow. So in the code base, we are aware of that, hey, this particular label is attached to this particular arrow, and this arrow has this label. So we just need some relationship between these two elements. How should we render the label for multipoint arrows? I'll be discussing this in a while as well. And how should we render the label when we are rotating an arrow? And lastly, we want to support export to SVG as well. So we'll be diving a bit into SVG as well. So let's have this set of requirements. I'll be diving into each of it. And we'll be fulfilling it one by one and have a minimal prototype at the end. So before that, let's understand how we want to structure the elements in the Canvas.

So before that, let's understand how we want to structure the elements in the Canvas. This is the bare minimum JSON, which you can think of. So it has a type, which is basically the type of the element. Is it an arrow or is it a text? The ID of the element, which you will be using to uniquely represent the element on the Canvas. So every element on the Canvas will have a unique ID. The XY are basically the coordinates to position them on the Canvas. Width and height are the dimensions. Points is basically an important parameter when it comes to arrows because an arrow can have two points, 100 points, 1,000 points, who knows? So it's basically the array of those points. And start arrowhead and end arrowhead are basically the two extreme points of the arrow, and the shape of the arrowhead, you can decide on. You can have a star. You can have a circle, whatever shape you want to come up with. So this is like a minimal JSON. Let's consider this JSON.

So I just shared this in socials to understand the audience and asked them to fill a Google form just to understand what is the understanding of the audience for this talk. And this is what I came up with. So I got like 63% of the audience will be beginners and 37% will be intermediate and similar for SVG. So can I get a quick raise of hands? Who are our beginners in Canvas here? Okay. Awesome. Thank you. Thank you for participating. I think some of you participated in this, and thank you so much.

So yeah, let's actually go to the first requirement now. How can you render the arrows in Canvas? So let's get into the basics first. Canvas drawing API is broadly divided into two categories. One is rectangle and one is paths. Rectangle is for drawing the rectangles. Paths is basically a collection of a lot of points and with which you can draw any complex shape in Canvas. And in this talk, we are going to talk a lot about paths. So coming to the basics, you create a canvas. You get the context so you can interact with the canvas.

Before starting any path, you have to call the begin path API and then just set some stroke styles, which is the color of the arrow. Once you do this, we have move to and line to APIs. Move to basically means that you move the pen to x, y, which is shown in the diagram. And then I draw a line to x plus length, y, which is still not visible. To make it visible, to paint it on the canvas, you need to call the stroke API. This is like the bare minimum way you can actually draw a line.

Now let's draw the arrowheads. So I am at this particular point, the extreme end of the line. From there, I actually draw one half of the arrowhead. So you can see on the top left, I have zoomed in the arrowhead, which is 20 pixels of width and 10 pixels of height. So I basically draw a line to A, B, which is the top coordinate. And then again, I come back to the same point, x plus length, y, and then again I draw one more line to A, C. That is how you actually draw one arrow. It might look like so many steps, but it's just a combination of line to end, move to API. That's all.

So, yeah, this is about the arrow. Now let's look at the multipoint arrows. A multipoint arrow can have more points, more than two. So this one has four points. And let's see. Let's quickly go through how this will look like. You basically draw a line, draw this line from x to x plus width comma y, and then just call the stroke API so it's visible. Then you draw another line, call the stroke API, draw another line, call the stroke API, and then just draw the arrowheads. So in this one, if you notice, we are calling the stroke API multiple times. This can actually, and for, just imagine if the arrow has 100 points or 1,000 points. And if you call the stroke for each and every line, this can actually be expensive. And so what we should be doing is we should just batch all these calls and call the stroke API only once because that will also give you the same result. So I just have a quick demo to show you how the performance can be impacted in these cases.

So, yeah. So this one is basically on the left side. I'm basically drawing an arrow that I'm calling the stroke multiple times, as you can see here. I'm basically calling this stroke inside the loop. And let me just complete this function where I just call the stroke API just once. So let me just copy it here. Single polyline, by single polyline, I mean that I'll be calling the stroke API only once at the last. And since it's multiple points, it's polyline.

And now I just copy this. And now let me just uncomment this one. So... Yeah. So now let's see how it behaves. So let's see. Let me just do it. Let me just refresh it once. Yeah. Let's see for 100 points. If you see, it's like single polyline is taking .2 milliseconds versus multiple is taking .3 milliseconds. So 1.5x faster. For 1,000, it's 5.9 versus 5.8. Not a very big difference. But if I go beyond this, I cannot show you in my machine right now, because my browser can crash because it's a very expensive API. So I just have a quick recording for that, just to make you understand how expensive can it turn out to be. So let's just go through that.

So let's just go through this quick demo. So for 10,000, you can see it's 228x faster. And then just my browser crashes. So this is what I was pointing to, that if you call the stroke API multiple times, this is what will happen. If you see this one, it's .7 milliseconds versus 159 milliseconds when you are calling the multiple times.".

And after this, my browser will just crash. And it depends on where are you testing it. I'm testing it on Chrome and Mac. It might differ when you test it on different browsers. So yeah. So yeah, that's something to keep in mind, that use this cautiously. Don't unnecessarily call the stroke API for complex paths. Just call it once. Or batch the calls.

In Excalidraw, we don't have to use this low-level canvas APIs. We use RoughJS, which gives you a sugar-coated APIs to draw the shapes. And additionally, if you have used Excalidraw, you will see that hand-drawn feeling, the strokes, the roughness. All of them actually come from this RoughJS library. It's a fantastic library by Preet. Do check it out if you want to. And yeah, it's a great library. So yeah, do check it out.

And now, the first requirement is fulfilled about rendering the arrows in canvas. Now, let's go to the next requirement, which is we want to render the label on top of the arrow. So let's get to the basics first. How do you render a text in canvas? You take a text, give it the font and the fill style. And the fill text API is basically which you will be using to render the text in the canvas. Now, here, I have taken one simple text, hello react summit with a smiley. And it is rendered like this with a pinkish color. Let me take a multi-line text. And this one, if you see, I have slash n in my text. But when it is rendering, it is rendering as a single line. Now, this is because canvas doesn't support multi-line. If you have multiple lines, you have to split them individually and you have to render them individually in the canvas.

If you have multiple lines, you have to split them individually and render them individually in the canvas. To do this, split the lines by slash n and render them with a difference in line height so they are not stacked on top of each other. Line height is a concept from the browser, with recommended values of 1.2 or 1.25 for unit-less line height. Since canvas works with pixels, multiply the line height by the font size and render each line individually.

To clear the overlapping area between the arrow and the text, there are two approaches: using the clear rect API or clipping. To use the clear rect API, first calculate the dimensions of the text using the measure text API. For multiline text, split it into lines and calculate the width of each line, taking the width of the longest line. The height is calculated by multiplying the font size by the line height and the number of lines. Once you have the dimensions, call the clear rect API to remove the pixels from the canvas. Then, you can render the text on top of it.

So we want to actually clear this overlapping area which is with the text. That is something which I want to clear so that it looks like the arrow is actually part of the text and not really some overlapping element on top of it. So how can we do that?

So for that, we want to basically clear this particular area from the arrow. And for that, we have two approaches. One is using the clear rect API and another is clipping. So let's see how the clear rect API looks like. So for that, first I need to calculate the dimensions of the text. And to calculate the dimensions of the text, I can use the measure text API from the canvas. Measure text API is basically provided by canvas and it gives you a lot of other metrics. Width is something which I'm interested in. So I can just use this API to calculate the width of a given text. And height, as I mentioned, it's just font size into line height. That is the height of the text. But what about multiline text? Because canvas doesn't support multiline. So the measure text API will also not work with multiline text. Again, we have to split it into lines, calculate the width of each line, and take the width of the longest line. In this case, page when looking at it has the longest or the largest width and that is the safest width to take. Similarly for height, just do the font size into line height and multiply it with the number of lines, whatever lines you have. And that will give you the height of the text. So yeah, once we have this, we just call the clear rect API and we just and then with the help of this API, these pixels are removed from the canvas and you can just render the text on top of it and this is how it looks like. This is what we want to see. So yeah, this is basically the implementation with clear rect. It's simple enough to start with.

Now let's look at the clipping as well, how the clipping works. So let's understand what exactly is clipping. Let's consider this rectangle on the right side and I want to basically clip this particular region which I'm showing with the black rectangle.

Now so basically what you can... So there is this rect API which you can call when dealing with clipping. So what we do is just draw this rectangle and call the clip API and only this portion of the whole drawing will be visible, whatever part you have clipped. Now this was a simple clipping region, just a small rectangle.

I can make it more complex by combining multiple parts. So let's say I want to clip this region where you have two rectangles. And in this case, again if I call the clip API, only this particular region of the whole drawing will be visible because this is the clipping region. In addition to this, the clip API takes one more parameter which is... Which basically helps you... So basically another parameter is even odd which basically creates a hole at the intersection region of the clipping parts. And this is interesting because this is something which we can actually utilize when it comes to labeled arrows. So this one, what we can do is since we know that, okay, we take multiple clipping parts and wherever it is intersecting, just a hole is created, means pixels are removed from there. So what we can do is let's take the outer rectangle because this area we want to clip, we want to make this entire arrow visible. Let's take another rectangle which is the text dimensions. And this particular region where these both are overlapping or intersecting, a hole should be created. The pixel should be removed. That is what we want. So once that is done, again, you will have the same output which you had with clear rect. These pixels will be removed and you can render text on top of it. So this was about clipping.

Now the performance varies in both cases again. So let's actually go to the quick demo where we can compare the clipping and the clear rect performance.

So in this case, right now the clipping is not yet implemented. So let me just quickly implement it here. I'll just... For the purpose of showing that how it is going to look like, let me just do this. Yeah. So basically this is the outer rectangle just to show you that this is something which I am going to clip. And then let me also draw the inner rectangle, or the text dimensions. Maybe I'll give it a yellow color. And then let me do this. And let me just take the text dimensions and see how it looks like. So, yeah. As you can see, this is basically the... The yellow part is basically the text dimensions which I'm going to clip. And the black is the outer rectangle. And wherever these two regions will intersect, a hole will be created. So let me now remove this part and actually use the clipping one which I already have here. And once we do this, you can see that the pixels are removed and it's same as clear rect.

Now let me just run through it and see how the performance looks like. For the first time, it looks like clipping is faster. It takes 0.1 millisecond for just one iteration versus 0.4. Let's see for 10. Let's see for 100. Now it gets a little slower. 5 millisecond versus 2.4. And let me do for 1000 iterations. So as many times I'm increasing the iterations, as many times I'm rendering this again and again, the clipping gets a little slower. 1.11 millisecond versus 15 millisecond. I can't go beyond this.

My browser will crash because clipping is also an intensive operation. So let me just show you a quick demo how slow it can be on my machine. So let's see here. So if I go to 100, I go to 1000. And let's see how it behaves for 10000. Yeah. So if you see here, 12,551 millisecond versus 120 millisecond. You can imagine how slow the clipping has become when you have 10000 elements or 10000 iterations, like 10000 elements you can imagine because I'm repeating this operation 10000 times and then my browser is crashing.

Now once you have rendered the text and clear rect looks like faster. That's what we use in Excalibur as well. Now what we have to do is so that users can actually interact with this feature, we need some DOM on top of Canvas. So just render a text area on top of Canvas, a position absolute so it's placed on top of it. Remove all the styling. As you can see, margin 0, padding 0, border 0, outline 0, so that the users feel that they are actually typing in the Canvas, but under the hood it's a DOM element. So it's basically to make users feel that they are actually typing into the app, typing into the Canvas directly.

Now I want to link the label and the arrow so that as developers we know that which label is attached to which arrow and which arrow has which label vice versa. So let's have the some assumptions for this to keep it simple. One arrow can have only one label. Fair enough. I don't think we would want to attach multiple arrows or multiple labels to one single arrow or maybe then we can discuss, but yeah. A label can be attached to only one arrow. So it's basically a one-on-one relationship between the arrow and the label.

So let's go to the third requirement. Now I want to link the label and the arrow so that as developers we know that which label is attached to which arrow and which arrow has which label vice versa. So let's have some assumptions for this to keep it simple.

One arrow can have only one label. Fair enough. I don't think we would want to attach multiple arrows or multiple labels to one single arrow or maybe then we can discuss, but yeah. A label can be attached to only one arrow. So it's basically a one-on-one relationship between the arrow and the label. And this simplifies a lot of things.

Now what we can do is in this JSON, what I want is by looking at this JSON, I should be able to identify whether it's a regular arrow or it's a labeled arrow. Since every element has a unique ID, let's just add a label ID to it, which will help me to identify if an arrow has a label ID, it means it's a labeled arrow. Otherwise, it's a regular arrow. Same thing for the text. If a text has a container ID, it means that it's a part of a labeled arrow and not a regular text. Otherwise, it's a regular text. Linking both of these is fairly simple.

If a text has a container ID, it means that it's a part of a labeled arrow and not a regular text. Otherwise it's a regular text. So yeah, this is this looks simple. Linking both of this is fairly simple.

Now once we have done this, what we can do is as I'm moving the labeled arrow, I can check whether this arrow has a label ID. Let me move the text as well. When I'm resizing the labeled arrow, I can check does this arrow have a label ID? Let me resize the text as well. So whatever actions you apply to the labeled arrow will be replicated to the text element as well because you are able to link both of these. So yeah, once this is done, this requirement also gets fulfilled.

Now let's jump to the next one, which is how should the label be rendered for multi-point arrows? So in this one, if you see, these are multi-point arrows. By multi-point arrow, I mean any complex arrow more than two points. So in this one, if you see, by following the older logic, where I took two extreme points and I render the text in the line, the midpoint of the line, then it would look like this because this is the line by joining the two extreme points of the arrow and this is the middle point. But this looks a little disconnected from the shapes, right? It's not really connected to the shapes. So now we need some intuitiveness, some logic here, how we can make it better.

So what we can do is an arrow can either have odd number of points or even number of points. What we can do is this is an arrow with odd number of points. So when you have odd number of points, you always have a middle point. So let's just attach the text to the middle point. Wherever this middle point is, let's just attach the text to the middle point. That way it will be intuitive for one part of the arrows where you have odd number of points. Now what do you do? This is how it will look. So now what do you do when even number of points, you don't have any midpoint here, but when you have even number of points, you have odd number of segments. So if you see one, two, three, four, five segments are here, let's take the middle segment and middle point of the middle segment. So in this one, we can just render the text in this particular position where you have even number of points. So in gist, this is how it will look like. Now the text is actually attached to these arrows and it will make you feel that, okay, these multipoint arrows are actually connected with it. So this is one way of doing it. This is how it's done in Excalibur as well. And the next one is how we want to render the label when the arrow is rotated to a certain angle. Let's consider these example.

The left side, the text is not rotated or text is rotated. The right side, the text is not rotated. Which one do you think is more readable, L or R? Yes, R is more readable. Great. Which one do you think is more readable, L or R? Can I hear more noise? Okay. So, in both the cases, when the text is not rotated, it's more readable, right? So this is how it looks like if you compare the text is rotated and text is not rotated. And this is how, if you see, if you use Excalidraw, you will see whenever you rotate the arrow at a certain angle, the text will never be rotated. We will always render it from left to right. And this is the reason because we want to give the importance to the readability. But, yeah, if we get future requests to rotate the arrow text as well, then we can consider it depending on the amount of request. But, yeah.

Now the last part. How do you actually export the labeled arrows as SVG? This is also something which we want in our it is it was in the list of requirements. So let's go into it. Let's get to the basics first. This is how the SVG looks like. SVG is basically vector-based. It's in the DOM unlike Canvas. You use this path elements to render the arrow. If you notice here, M basically stands for move the pen to, similar to the move to API. L basically stands for draw a line to 200 comma 10. Basically similar to Canvas line to API. So it's basically the same API, just a different way of writing it. So let's relate these to each other. Now if you can visualize all the move to line to APIs are actually converted to SVG APIs. And that's like that's the conversion. But it helps you to visualize it, how it basically both are the same. It's just a different way of writing it. So now, again, we are at the same problem. How do you actually hide this part of the arrow in the SVG? So we want to hide again this part in the SVG as well where the text and the line are We want to mask this particular area.

Now when it comes to masking, white means reveal the area, black means hide it. White means transparent, black means opaque. This is a rule to remember. So let's remember this rule. And let's see what masking looks like in SVG.

Again, it's the same red rectangle which I showed you a couple of minutes back. And this is how you can add masking. You basically add the masking attributes, add some ID to it. And then, as you can see, in the rectangle, I'm applying that mask with the help of the mask ID.

So let's add some masking regions here. Now if you see, I have added four masking regions here. And each have a different background color. Each have a different fill color. So if you notice, the second rectangle, the second masking region is fully invisible because it's filled with black. Black means opaque. And the fourth one is fully visible. And the last one, it's fully visible because it's filled with white. White means transparent. So when you fill it with black, it basically hides those pixels. And when you fill it with white, it basically makes it transparent.

So let's use the same logic when it comes to labeled arrows. Let me draw an outer rectangle because I want to make this complete region visible. So I make it transparent, I fill it with white. And the text dimensions, I want to hide this particular region from the arrow. So I will fill it with black. This is only happening in the arrow SVG, the text is outside the masking. So if we do this, basically the black area should be hidden or removed from the SVG. So let's see, this is how it looks like. Now let me actually add the white background rectangle to the mask. And when I do that, you see the arrow is visible because it's filled with white.

White means visible or transparent. And if I add the black one to the masking region, that area is actually removed. So this is how you can actually use the masking in SVG to hide this overlapping area when it comes to labeled arrows. And yeah, that's how it works in SVG.

Since we fulfilled all the requirements, let's discuss some performance improvements to enhance this feature and make it faster. It's recommended to draw a single polyline instead of multiple lines when drawing multipoint arrows. Using an offscreen canvas, which is not rendered on the DOM, can improve performance. Instead of rendering shapes every time, you can render them on a temporary canvas and take a snapshot using the Draw Image API. This significantly improves rendering speed as demonstrated in the quick demo.

For any repetitive tasks in canvas, try moving them to an offscreen canvas or temporary canvas. Additionally, rounding off workload point values is important to avoid pixel rendering. These best practices can greatly optimize performance in canvas applications.

So let me just show a quick demo for this one as well. So in the left side, I am rendering the text multiple times. I'm calling the fill text API again and again. If you see here, I'm calling the draw text, which actually calls the fill text API again and again. On the right side, I'm actually using the Draw Image API. So I'm actually taking a snapshot of it and actually copying it to the main canvas. So for 100 times, when I'm rendering this text 100 times, you can see that it is drawing images taking 1.3 milliseconds versus when rendering multiple times, it takes 2.3 milliseconds. Let's do it 1,000 times. It's 15.2x faster, the Draw Image API. Let's do it 10,000 times. Now this difference looks good. 8.2 millisecond versus 177 millisecond, which is huge. So yeah, it's not just for text or arrow. Any repetitive tasks, when it comes to canvas, try to move it to offscreen canvas or temporary canvas. Workloading point values, because it leads to some pixel rendering. This is always, you would have seen this mentioned in the docs as well. So always try to round it off.

This is an interesting one. Rendering arrow in parts can help in improving the performance. What I mean by rendering in parts. I draw the part of the arrow before the text, then draw the text, and then draw the rest of the arrow. So you are dividing the task into three steps instead of using the clear rect or those kind of things. And this actually can improve the performance. Let me just quickly show you. So the left side is with clear rect. And on the right side, I'm actually drawing it in parts. And if you see here, I'm actually drawing the first part of the arrow, then the second part of the arrow, the first part of the arrow, then the text, and then the second part of the arrow. So for the first time, it's super fast. Let's do it for 100 iterations. It's 1.9 milliseconds versus 8.5 milliseconds.

Drawing in parts is faster compared to clear rect. 7.7 millisecond versus 89.2 milliseconds. Again, this is a great improvement. So, yeah, this is something which you can actually try out. But at the same time, this can be complex to implement, depending on the type of arrow, because you have to figure out the points up to which you should be rendering and then you render the text, and then you render the rest of the arrow.

So yeah, wrapping up the talk, let's look at the takeaways, what you have learned so far. We learned about rendering the labeled arrows in Canvas. Canvas doesn't support multi-line, so it needs to be handled when rendering text. Clear rect versus clipping. Clear rect is a lot faster than clipping. Use off-screen Canvas to draw the text and the arrow, or any repetitive tasks, and copy to the main Canvas. Drawing single polyline is always efficient than drawing multiple lines. Avoid floating point values and round it off to integers to avoid some pixel rendering. We also learned about labeled arrows, rendering it in SVG, and use masking to remove the overlapping area. And yes, drawing arrow in parts is fast versus clear rect, but it can turn out to be a lot complex when it comes to the implementation. We don't do this in Excalator, but we might do some part of it for two-pointer arrows.

So yeah, with this, this is all the demos. If you want to try it out in your machine and share how it works for you, because every device browser will have different requirements, so it might change. But if you want to try out, you can actually scan this QR code and try it out. And yeah, these are my socials again. And yeah, lastly, I'm an independent open source developer, so please support me so I can keep contributing to open source. That's all from my side. Again, ending with the cycle theme. So yeah, thank you so much. Thank you to the organizers for having me, and yeah, if you have any questions or any feedback, do let me know. I don't think I'll have time for questions, but we can take it outside. Yeah? OK, thank you.