Computer Vision on Your Browser With SVG Filters

Hi everyone, my name is Adam Klein. I'm the CTO and one of the co-founders of Cover with Double V. We are a swag store platform so we can open swag stores, send branded gifts globally to your employees to celebrate different milestones. And it's a very cool platform and we do a lot of computer vision on the browser. That's how we got to know SVG Filters and this is how this talk was born. So let's get right to it.

I'd like to start this talk with two magic tricks because one of my secret passions is magic and I especially like to know how magic is done. So the first magic trick I'll do a very simple card trick. This is a branded card that you can order on our platform of our company or of your company and the trick is very simple. We just take the card and we, sorry, make it disappear. And then we make it reappear out of thin air. This is the first trick and I'm going to show how it's done in the end of the talk.

And the second trick is my favorite types of tricks is a trick that was already made and you weren't even aware of it. Because as you can see, I'm recording my webcam and this is in fact not a Zoom filter or a background. This is just a green screen filter that I created with this simple SVG markup code. And by the end of this talk, I'm going to teach you, you're not going to know how this code works and as you can see, it's super simple. So let's start first from moving to better background than this ugly green screen filter. Perfect.

So let's learn about SVG filters. As you can see, I've made this small like Instagram app where you can change filters. So the first one you saw was a green screen filter. We have other filters like this cool frosted glass effect or this infrared spooky kind of effect or this with the like there are water drops on the screen and there are other filters as well. And the cool thing about SVG filters, as you can see, it's just SVG markup. There is no JavaScript, there's no it's just SVG. It's a primitive computer vision, simple computer vision primitives that you can combine together to create these cool effects. Now because it's all SVG markup, you can apply it on whatever you want. So here I'm applying it on the webcam. I can apply it on an image or sorry on a video. I can apply it on as you can see there's a video on the background. I can apply it on an image.

I can apply it on text. Hello. Again, a different filter, frosted glass on an image on a video. And so this is really cool. And because again, this is just HTML or SVG, it's really easy to animate.

So let's take this water drops effect. It's really cool to animate it and make the water drops drop. Let's go back to the webcam so you can actually see me. So as you can see the waters are dropping, it's really easy to do. Sharpen effect, so this is without the effect, this is with the effect. Very cool. This cool diffusion animation effect, pixelated right again on an image on text and this really cool crayon effect. So this is all done with very simple SVG markup code, very different effects and I'm going to show you how it's done. So let's start from this frosted glass effect and let's turn out how this is done.

So as you can see, this is a filter. It has an ID and this filter can later be applied on different elements. And the filter is like a pipe of functions that run on the pixels themselves. So as you know, an image is just an array of pixels and every pixel has a value, a red, green, blue and alpha, which is the transparency. So let's see for example how the frosted glass effect works. And what's really cool about SVG filters, you can easily debug them and break them down because they're just a markup code. So let's comment out these two and see how the first filter looks like. So this is the turbulent filter. As you can see, it's just random noise. So and you have different ways to define how the noise is being made. We can play around with the parameters but as you can see, it just creates random noise.

Okay, how does this help us? Let's take the second, actually the last one here, which is the displacement map. And now you see, okay, suddenly somehow this noise is being translated to the element itself. And the displacement map is a very simple primitive. What it does is displaces pixels either by the x-axis or the y-axis. So this is the original image. And after we move pixels around to the left or right, up and down, everything gets this kind of weird. And the way that the displacement map works is very simple. We have, as you can see, x-channel selector and y-channel selector. What this means is we take the turbulence that we created before, this random noise, each pixel in the turbulence has a red value, a green value, blue and alpha. Here we ignore the blue and the alpha and we use the red and the green as a sort of meter to say how much do we move the pixel. So let's say if red in the turbulence, in the noise is zero, then we move the pixel a lot to the left. And if the value is 255, we move the pixel a lot to the right. And if it's 127, then we keep it in place. So basically we're using the red and green values just to tell the displacement map how much to displace on the x-axis and on the y-axis. And scale basically means how much do we move it. So if we return the frosted glass effect, if we increase the value here, suddenly the effects gets more intense. If we decrease it, then it's much lighter. But it's still not exactly the frosted glass effect that we had before, because it's more smooth because the turbulence creates smooth transitions between colors. The way to make it like this frosted glass which is more broken effect, we had another filter called a component transfer, which allows us to run a function on each pixel. Instead of moving it around, we change the value of the pixel. So the pixel stays in place, but we can change the values of different channels. So let's return the effects and return this filter. And as you can see, now it's more broken because what we did is we ran a function called discrete. So in component transfer, you have different functions. Here we chose a discrete. We run it on the red channel and the green channel. And we tell it, OK, it doesn't matter what value you have here. The values are between 0 and 1, not between 0 and 255. And we say it's kind of like snapping. So it snaps to the nearest value out of this list.

So the output would be either exactly 0, exactly 0.6, or exactly 1. So if it's 0.9, it will snap to 1. If it's 0.7, it will snap to 0.6, et cetera. So it puts discrete values. That's why this effect is more broken than smooth.

Cool. So this is the frosted glass effect. Let's look at another one, which is the infrared effect, which is one of my favorites. And again, as you can see, really simple code. Turbulence and displacement you already know. The main difference that we did here is, let me show you the turbulence. Instead of this completely random noise, it's still noise. But as you can see, I gave it these different values of frequency for the x and y axis. That's why you see it's more like these lines instead of dots. And that's why it creates these edges, this disturbance in the image.

And the next one, which you still don't know, is the column matrix, which looks like, if you saw this code anywhere, you would say, I have no idea what these numbers are. They're just random numbers. But after you hear the explanation, you will know it's pretty easy. So a column matrix, what it does, it takes every pixel in the input image and multiplies it by the matrix. And now you're thinking, oh no, matrix multiplication. I learned this in my computer degree. It's actually really simple. So let's just copy it here so I can explain it. So basically, and let's just put me back on a regular webcam. So basically what this means is every line here just calculates a different channel. So the first line will calculate the red channel. The second one will calculate the green, the blue, and the yellow. So this will be the output pixels. And the columns are just the input pixels. So this is the red.

Let's just comment this. Sorry. So this is just the red, the green, the blue, the alpha. And here we have some constant that we had. And this is just a linear function. So the output red will be 1 times the input red plus 1 times the input green plus 1 times the input blue, RGB, plus 0 times the input alpha plus 0. So what does this mean? The value of the red in the output pixel will take its value from the intensity of all the input colors. So if the original color is very green, then the output will be very red. If the original color is white, which has 1 in all of the elements RGB, then it will be a really intense red.

What about green and blue? We just make them zero. So there's no green component, no blue component. And alpha is just the identity equation. So the output alpha equals the input alpha. So again, after you realize this is just outputs and inputs, very simple linear functions, then you get this effect. And as you can see, like my shirt was white, now it's very red. My beard is black, so it's very dark. And you can play around with it. So if you wanted, you know, if I just zero down all of this and made it only blue, then we'll get everything blue. It's really simple. Or if we make this one, everything is just red, because if it's above 1, it will be 1. And 1 is just 255.

Okay, so now I think it's time that we reveal how we did the green screen filter. And I think what we're going to do is we're going to build it together with you. So let's copy this maybe here. Okay, so the first thing we're going to do is we don't want displacement, right? We want all the pixels to remain where they are. And we don't even want to change the colors. What we want to do is to say, if a pixel is green, we want to make it transparent. Why? Because in the background, there's something else, let's say an image or a video. So if the pixel of the webcam is transparent, then you can see what's behind it.

So if the pixel of the webcam is transparent, then you can see what's behind it. So let's start with a very simple color matrix, where the values are the identity metrics. So the identity matrix basically says every input pixel is mapped to the exact same pixel in the output. So not a really interesting equation, it just takes the same pixel and maps them. I'm looking for something green to show you, but I'll find it later.

This basically does nothing, which is not what we want. We want a green screen filter. But what we want is just to affect this. This is the alpha. So for example, if I make this zero, then you can see everything is fully transparent, which is not what we want, right? We want everything to be opaque, except for the green pixels. So what we can do is say, okay, if a pixel is green, let's reduce the opacity. How do we do it? For example, we can take the green pixel and multiply it by minus one. And now we get this weird effect where everything is a bit darker, sorry, more transparent. And that's because we have a smooth function, right? If the input pixel is one in the green, then we'll have minus one plus one, which is zero, which is fully transparent. But most pixels have, you know, a value somewhere between zero and one.

So what we want to do is add the component transfer, a discrete component transfer like we had before. And we want to tell it, okay, the values should be either completely opaque or completely transparent, which is basically what we want with the green screen filter. But what you see now is like my shirt is white. Why is it doing this filter? Because white has a very strong component of green inside it, right? White is just 255, or one green, one red, and one blue. So what we really want to say, we don't just want to take pixels with a strong component of green. We want to take the pixels with where the component of green is much stronger than the components of red and blue. So what we can do is say, okay, if the pixel is red, let's add opacity. If the pixel is green, let's remove opacity. And if it's blue, let's add opacity. And I'm going to get something green just to show you. So as you can see, this is green. But we don't see any effect. Why? Because red and blue are winning. And we have this one, like we start with a base of one.

And we have this one, like we start with a base of one. So let's increase this to 2. Okay. And now you can see green is becoming more transparent. But if I showed you a few things, then you will probably start to see false positives. I don't know if you can see it on this. But you start seeing false positives if you have things that are almost one or almost green. So what you can do is start playing with this number. So like if this was, for example, if this was, for example, minus three, then suddenly you get much more false positives. But if it was minus, let's say, one and a half, then you have much more false negatives.

So basically what I did after playing around with the numbers a lot, I got to this formula, which is what we had before. And I also added more weight on the one in the discrete function. So basically I'm preferring one over zero. And I got this function and it works. I tried it on different backgrounds and it worked great. All right. So that's it. Now you know how SVG filters work. I heard quite a few companies that are actually using them in production. They are supported in all major browsers. And as you can see, they're very performant. They can work on the webcam because they run on the GPU. So they're amazing.

And one of the things you'll notice when you start working with SVG filters is that it's really hard to understand from the documentation what you can do with it. So by reading the documentation, you won't understand how to build an infrared filter or frosted glass or whatever. You have to be kind of creative about it. You have to play around with it, see what kind of effects you get. It's not the usual mindset when you develop something that the product tells you, this is what I want, and you think how you do it and you implement it. Here it's much more experimental. It's much more creative. It gives you as a developer much more creative spirit, which is why I love them the most. And as you can see, now you know how the magic is done. I hope you enjoy it and you use it and you enjoyed the talk and you learn something new. And what I really like about knowing magic tricks is something that seemed impossible before now seems very easy. And I hope you feel the same. And let me know if you have any questions. Goodbye.