This is just 3 megabytes in size and can recognize 468 facial landmarks on the human face. And this is a really cool feature, and I'm going to show you how to use it on the slide right now.
Now, people are starting to use this for real-world use cases, such as L'Oreal, and they've created AR makeup try-on, which allows you to try on lipstick in this case in realtime without even having to be there physically present in the store. So you should note, the lady on the right-hand side is not wearing any lipstick. We're using face mesh to understand where her lips are, and how to apply the lipstick she wants on her face in realtime. So this is super cool, and I'm sure we're going to see more stuff like this coming out in the future.
So let's see face mesh in action to see how it performs in the real world. Let's switch to the demo. Okay, so now you can see me talking to you with face mesh running in realtime in the web browser at the same time. On the left-hand side here, you can see the machine learning in action, and there is indeed a mesh of my face being overlaid on the web browser, and I can move my face around, and it's pretty robust. I can open and close my mouth and my eyes, and you can see that happening all in realtime. And we're getting a pretty solid 25 frames a second or so as I'm running on my GPU via WebGL.
Now, of course, I'm also streaming this live, so I'm using some of my GPU resources right now. I could get higher frames per second if I desired, but I can also switch as a developer to Wasm, which is WebAssembly, to execute on a CPU, or I can choose to execute on the CPU by itself, which is the slowest form of execution. So there's a number of options on where you can deploy to. And because this is JavaScript, not only am I doing the machine learning there on the left-hand side, I'm also able to render this 3D point cloud of the results on the right-hand side using 3.js. And JavaScript from day one has been designed for the presentation and display of information, so you've got a lot of libraries, especially for 3D graphics, to be able to do this very, very easily. And you can see I can move it around and now inspect my face from different angles, which is being constructed in realtime live in the browser.
Okay, back to the slides. So next up is body segmentation. This allows you to distinguish 24 body areas across multiple bodies all in realtime. You can see this is an action on the slide, and on the right-hand side, we've got a number of lines that represent different parts of each body. Even better, we've got pose estimation going on at the same time, those light blue lines contained with each one of the bodies on the right-hand side there, which allow us to estimate where the human skeleton is. And with that, that can enable really powerful demos, such as the ability to recognize when you're in a certain pose, or a gesture, or something like this. And we've got a lot of great data from our yoga instructors and this kind of stuff, so it's super cool to see the creative potential of this model. And in fact, with a bit of creativity, we can use things like body pics in a number of delightful ways. Here's just two examples I created in my spare time. On the left-hand side, you can see me becoming invisible as I walk around on my bed, and notice when I walk on the bed, the bed still deforms. And with this static image, I'm able to calculate the background updates in real-time and only remove my body from the parts where my body actually is. And of course, it's not perfect, but it's the first steps, and this was made in just a day.
Comments