TensorFlow.JS 101: ML in the Browser and Beyond

Rate this content
Bookmark

Discover how to embrace machine learning in JavaScript using TensorFlow.js in the browser and beyond in this speedy talk. Get inspired through a whole bunch of creative prototypes that push the boundaries of what is possible in the modern web browser (things have come a long way) and then take your own first steps with machine learning in minutes. By the end of the talk everyone will understand how to recognize an object of their choice which could then be used in any creative way you can imagine. Familiarity with JavaScript is assumed, but no background in machine learning is required. Come take your first steps with TensorFlow.js!

This talk has been presented at JSNation Live 2021, check out the latest edition of this JavaScript Conference.

FAQ

TensorFlow.js is a library for machine learning in JavaScript, allowing users to run, retrain via transfer learning, or write their own models from scratch. It can be applied in web browsers, on the server side, and across various platforms like mobile and Internet of Things (IoT). Common use cases include augmented reality, sound recognition, sentiment analysis, and more.

Using machine learning in JavaScript with TensorFlow.js is powerful because JavaScript can run across various environments without additional plugins, such as web browsers, servers, desktops, mobile devices, and IoT devices. This makes it highly versatile and accessible for integrating machine learning into a wide range of applications.

Real-world applications of TensorFlow.js include object detection, body segmentation, facial landmark detection, and pose estimation. Companies like L'oreal use it for augmented reality features such as virtual makeup trials. Developers can also create interactive applications like body size estimators and augmented reality effects.

The advantages of using TensorFlow.js on the client side include preserving user privacy as data does not need to be sent to a server, reducing latency for faster processing, lowering operational costs, enhancing interactivity, and leveraging the widespread reach and scalability of web technology.

Yes, TensorFlow.js provides access to various pre-trained models that are easy to use for common tasks such as object detection, face landmark detection, and natural language processing. These models can be easily integrated into applications without needing to train your own models from scratch.

TensorFlow.js supports retraining models through transfer learning. This can be done using tools like Teachable Machine, which simplifies the process of training and inference completely in the browser, or more robust systems like Google Cloud AutoML for training custom vision models with larger datasets.

TensorFlow.js models can be deployed across various platforms such as web browsers, servers, and Node.js. For web applications, models can be executed directly in the browser using WebGL for GPU acceleration, or on the CPU. For server-side implementations, TensorFlow.js can utilize Node.js to leverage TensorFlow's CPU and GPU bindings similar to Python.

Jason Mayes
Jason Mayes
39 min
11 Jun, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

JavaScript with TensorFlow.js allows for machine learning in various environments, enabling the creation of applications like augmented reality and sentiment analysis. TensorFlow.js offers pre-trained models for object detection, body segmentation, and face landmark detection. It also allows for 3D rendering and the combination of machine learning with WebGL. The integration of WebRTC and WebXR enables teleportation and enhanced communication. TensorFlow.js supports transfer learning through Teachable Machine and Cloud AutoML, and provides flexibility and performance benefits in the browser and Node.js environments.

1. Introduction to Machine Learning in JavaScript

Short description:

Hello, everyone. My name is Jason Mays, I'm developer advocate for TensorFlow.js here at Google. We can use machine learning anywhere JavaScript can run, and that's actually a lot of places. JavaScript is one of the only languages that can run across all of these environments without additional plugins. And with TensorFlow.js, you can run, retrain via transfer learning, or write your own models from scratch if you choose to do so. Just like you might be doing in Python right now, but in JavaScript. And that means you can make anything you might dream up. From augmented reality, sound recognition, sentiment analysis, and much, much more.

Hello, everyone. My name is Jason Mays, I'm developer advocate for TensorFlow.js here at Google, and I'm excited to be talking to you about machine learning in the browser and JavaScript. So, let's get started.

Now, first off, why would you want to use machine learning in JavaScript? That's a great question. And if we look here, we can see that we can use machine learning anywhere JavaScript can run, and that's actually a lot of places. The web browser, server side, desktop, mobile, and even Internet of Things. And if we dive into each of these areas a bit more, we can see many of the technology we know and love already. The common web browsers on the left-hand side, Node.js, React Native for mobile native apps, Electron for desktop native apps, and even Raspberry Pi for Internet of Things. And JavaScript is one of the only languages that can run across all of these environments without additional plugins. And that alone is very, very powerful. And with TensorFlow.js, you can run, retrain via transfer learning, or write your own models from scratch if you choose to do so. Just like you might be doing in Python right now, but in JavaScript. And that means you can make anything you might dream up. From augmented reality, sound recognition, sentiment analysis, and much, much more. It's really up to you what you create.

2. TensorFlow.js: Pre-trained Models and Face Mesh

Short description:

Now, with TensorFlow.js, you can use pre-trained models for various use cases, including object detection, body segmentation, post estimation, face landmark detection, and natural language processing in the browser. Object recognition using Cocoa SSD is showcased, allowing for real-time detection and classification of objects in images. Webcam integration enables live object detection and classification, running client-side in a browser, ensuring privacy and cost savings. The three-megabyte face mesh model can recognize 468 facial landmarks, enabling real-time applications like AR makeup try-on without physical presence. See face mesh in action during the demo.

Now, with TensorFlow.js, we've got a number of different ways you can use us. The first way and the easiest way is to use our pre-trained models. These are super easy to use JavaScript classes for many common use cases, like we can see on the current slide. And here, we can see things like object detection, body segmentation, post estimation, face landmark detection, and much, much more. Even natural language processing is supported in the browser.

So, let's see some of these in action. Now, first up is object recognition. This is using Cocoa SSD behind the scenes and is trained on 90 object classes. You can see this in action on the right hand side with the dogs being highlighted with their bounding boxes. And we can even know that there's two dogs in this image as both are returned to us. So, let's see this in action live to see how it performs in the browser.

Okay, so here's a webpage I created that's running this code live in Chrome, and if I click on any one of these images, I can now get object detection working for any objects it finds in those images, even if it's different object class types. But we can do better than this. We can actually enable the webcam and then do this live in real time. And you can see me talking to you right now, and you can see how it's classifying both myself and the bed in the background sometimes as I speak to you. And this is super cool, because not only is this running at a very high frames per second live in JavaScript, this is running client side in a browser. So, that means none of this webcam imagery is being sent to a server for classification. Not only does that save cost, it also means my privacy is preserved, and that's a really big thing these days. And with TensorFlow.js, you can get that for free.

So, next up, we've got face mesh. This is just three megabytes in size, and can recognize 468 facial landmarks on the human face. You can see this in action on the left-hand side of the slide right now. Now, people are starting to use this for real world use cases such as L'oreal, and they've created AR makeup Trion, which allows you to try on lipstick in this case in real time without even having to be there physically present in the store. So, you should note, the lady on the right-hand side is not wearing any lipstick. We're using face mesh to understand where her lips are, and then we use WebGL shaders to augment the color of lipstick she wants on her face in real time. So, this is super cool, and I'm sure we're gonna see more stuff like this coming out in the future. So, let's see face mesh in action to see how it performs in the real world. Let's switch to the demo. Okay. So, now you can see me talking to you with face mesh running in real time in the Web browser at the same time.