Real-Time Robot Control From the Browser With WebRTC

Hi, my name is Nick Hare. I'm a senior developer advocate at a company called Veeam, a platform for connecting software developers with the physical world. You can find my handle and logo online at most places at Hipster Brown. I'd like to start out by saying I have no formal education in embedded engineering, robotics, or even software development. I've learned from others who have shared their knowledge through open source projects, presentations at local meetups and conferences, and online through videos and blog posts, which makes me so excited to be in a position to share that knowledge with you all today and hopefully inspire you to build your first robot.

Now, when I say robot, what image comes to mind? Growing up, I thought of robots like Rosie or C3PO and R2D2, or even Wall-E and Eve. These are examples of autonomous, intelligent, and sometimes humanoid machines that exist in imaginary worlds. Looking at the world today, there are some robots that match what we've imagined in various forms, atlas, digits, even figure zero one. But most production robotics is in the form of industrial arms or rovers, or really just various actuators and sensors combined together to perform some specific task to aid and assist humans in completing their jobs to be more efficient, safer, and do it from anywhere in the world.

And there are quite a few robotic functions that require monitoring and controlling those machines from afar. This falls under the category of teleoperations or tele-op. And this could include keeping an eye on an autonomous fleet of delivery vehicles or autonomous cars and taking over if they become lost or enter unexpected situations. It could be piloting vehicles that are in environments that are unsuitable for human life, such as deep undersea, in space, or the desert, or telepresence and telemedicine experiences. It can also be used to create unique art like the telegarden. Even those more advanced humanoid robots currently rely on a human in the loop to function.

Now, some robots communicate with remote servers in cloud environments. Some are confined to the local network. And others communicate directly with each other. But what sort of technology and protocols could support connecting to a machine from any other world without exposing it to the public internet? Well, as spoiled by the title of this presentation, we're talking about WebRTC. And you've probably heard about it before and used it in some application built with it.

Now, as the name applies, it was built around browsers and providing a platform for connecting to clients directly without needing to relay through some sort of central or shared server application, which might be required through using something like WebSockets. Due to its legacy tied to voice over IP and video conferencing, WebRTC is largely used for that exact function. We've all been through our fair share of Skype, RIP and Hangouts and Zooms and Teams calls over the years. Some of us may even gone through a peer-to-peer chat messaging tutorial in the past as well.

But those are really only the capabilities that WebRTC has to offer. What if a peer wasn't another human, but a machine answering the call? How could this be possible? Does the robot open a web browser and click around like some type of AI agent? Well, what if we could use the APIs and protocols of WebRTC typically found in the browser, but on a server, aka our robot? WebRTC is really just a stack of acronymed technologies that enable sending audio, video and arbitrary data directly between two clients or peers. And this is to say that most computing devices and programming languages have some way of working with the base protocols of IP and UDP.

And so the layers above just need to be implemented to support using WebRTC in those environments. And several projects exist to provide that today. At the relatively lowest level, there's the libpeer library written in C. There are a few options for Node.js and some mature projects for Python, Go and Rust.

Cool. So it seems like we can use WebRTC on the server, but how do we actually use it to connect to our robot? Well, for WebRTC to work, Stun is required to discover the IP address and port of each peer from the perspective of the public Internet. It's like using a GPS satellite to see where we are relative to the entire world rather than just what we see around us.

During this process, the client may also gather potential methods for connecting to each peer through Interactive Connectivity Establishment or ICE candidates. If they cannot connect directly to each other due to restrictions like firewalls, clients can fall back to using TURN or relay servers. Now, this is a last resort option as it adds latency to the connection, costs to maintain and a potential point of failure for the experience. During this process of gathering Stun and ICE information, two peers need the way of sharing that information with each other before a direct connection can be established.

And this is known as signaling. However, the spec itself doesn't require any details about accomplishing signaling. And so this is kind of a double-edged sword when trying to build a peer-to-peer application. So most of them will have some sort of server-side component that relays messages between the peers using web sockets or even plain HTTP calls. Whatever method or technology you use, there just needs to be some way for two or more peers to establish a connection and exchange that initial information for it to happen.

Once they're connected, then a shared data channel can be used to further negotiation in the process. So how does this work for a robot? Conceptually, we treat our server, like our robot, as an answerer ready to receive calls. A client, like our browser, is a caller. The signaling server matches callers to answerers so that they can exchange information and hope to establish a connection. So when a robot is active and online, it will let the signaling server know when it is available to receive calls. When a client application calls a robot through the signaling server, then the program on the robot will accept that offer, provide an answer, and listen to any data channels to be opened to establish those further communication patterns.

All right. So what do we do once we have that stream of data available? We could send arbitrary string messages, even stringify JSON, and make sure that our robot knows how to respond to each message. However, this would be quite tedious and error-prone as we build up both the robot logic and the web application. It would help to have some sort of structured format for sending commands from the client application to be executed on the robot without specifying, you know, what exactly they are or even just being able to specify the expected response for each command. Now, this is a perfect use case for remote procedure calls or RPC. Basically, this provides a specification of an agnostic way of communicating between servers and clients that creates a kind of facade that looks like you're calling a method on the client that gets executed on the server without really worrying about how that data gets there and back.

You might have heard about explicit RPC frameworks like gRPC, tRPC, and JSON-RPC, where the server and clients are clearly defined. Some implicit RPC usage is becoming popular for full-stack JavaScript applications called actions or server functions, and these are very coupled with HTTP. In our case, we want something that can adapt to our bidirectional data channel, and that's the stream coming from our peer-to-peer connection. Cool.

With all requirements laid out, let's build a proof of concept using full-stack JavaScript. We'll establish this and accomplish it with using mostly built-in APIs for the browser and Node.js along with a few packages. We'll break it down piece by piece before putting it all together, starting with signaling. Rather than build out and host a custom service to perform this task, I wanted to focus on just our peer code. So, I've opted for a good enough solution for PubSub messaging called Notify. This is an open-source service providing a REST API for sending and receiving messages around specific topics. No signup required.

Notify has a few programmatic ways of subscribing to messages via its API, using either HTTP endpoints or WebSockets. It might seem natural to use WebSockets, but they're really only used to subscribe to topics rather than the bidirectional connection that we typically use them for. So, this is a little bit more overhead than I'd like for my proof of concept. I'll stick with one of the HTTP endpoints to better suit my needs. Now, server-side events are a nice browser-native way of subscribing to messages over HTTP. However, I need something else for Node.js if I don't want to add yet another dependency. What about JSON streams?

The response object from making a fetch request can have a body property that implements a stream of bytes. And we can pipe that stream through some sort of transformation to convert it into UTF-8 text. And we can parse those JSON strings into the message objects that are defined by Notify. And we can wrap all this together in a sort of notifier class that provides all the different things around subscribing to messages, cancelling subscriptions, and even publishing them.

With our signaling method in place, let's take a look at establishing our WebRTC connection. So, we need our RTC peer connection class from either our Node.js package or globally in the browser. And if we only needed to connect to devices on the same network, there's really no need for stun or turn servers under this array of ice servers that we see when we're creating our new peer connection. But that's not very useful for a realistic demo. Only one of these stern servers is really needed. So, most of the applications will use the public Twilio or Google ones.

From that peer connection, we create our data channel for sending and receiving messages directly between each peer. This creatively-named data channel with the label of data is configured as a negotiated channel, where the agreed-upon ID is settled out of band. And which really means it's hard-coded into our program, since we know there should always be a single channel with this label. And this is where we diverge between our robot and our web application. As mentioned earlier, the client acts as a caller and the robot answers if it is online. So, the client creates an offer to connect. And this is set as the local description before sending it over to the robot through our signaling service.

And this is set as the local description before sending it over to the robot through our signaling service. Thankfully, most objects in this API have a two JSON method for portability across signaling services. The robot receives that message to set as its remote description before creating an answer to set as its local description and then sending that over to our web application. To complete this part of the exchange, the web app then handles our answer and sets that as its remote description. During all of this, we're starting to share and settle ICE candidates. So, the caller web application will listen for this ICE candidate event that indicates when a new candidate has been identified and added to the local peer connection. The caller will then send that information to the robot through our signaling service. And this can happen several times throughout the process, until our ICE connection state is connected. And if a candidate is null on the event object, then this indicates that the gathering of ICE candidates is complete. On the robot, it checks for this candidate in our message and adds it to the local peer connection. And this will signal to the internal process to attempt to connect using those details. Now, the peers will use all of this information to hopefully connect to each other and lead to an ICE connection state of completed. At the same time, this should align with an open event happening on our data channel to begin letting peers communicate over that transport.

With our WebRTC session in place, let's look at building our RPC framework around it. The JSON RPC 2.0 package provides our transport agnostic client and server classes for sending and receiving commands, as well as some TypeScript interfaces for those of us who want to ensure some consistency. We'll create a robot server and a robot client and use the data channel to send any RPC requests over that stream. The data channel can listen for new messages to parse as RPC requests and forward to our server and then other RPC responses to forward to the client. All right, and now we have our communication framework ready to go. Now, we only need to do is control our hardware. Johnny 5 started as a way of writing JavaScript to control various Arduino boards over serial port connections. It provides abstract classes to control all sorts of physical components in the same way that jQuery provided abstractions over inconsistencies in browser APIs. Since its inception, the core has been extended through IO plugins, extending to support all sorts of hardware platforms, including Linux based single board computers like the Raspberry Pi. All right, I believe now we have all the pieces that we need to control our robot in real time from the browser. On my desk to my right, I have a camera overhead of a Raspberry Pi with an LED attached to it. We can see in our web application, I have the ability to connect to a robot that I've called Robot Dev. Now, the server side of our process is running on my Raspberry Pi in Node.js, and this is running on GitHub pages in the browser. I can hit connect and we see we've connected to our robot. I can hit connect and we see we've connected to our robot. That's fairly quick and seamless. But we can see from here all the different messaging that happened in order to make that happen.

So we have offers. We have all of our signaling state happening, our ice gathering candidates. We can see we're sending them over to the server. And eventually getting some answers as well. Hopefully leading to our stable signaling states and connections, seeing that we've connected. And eventually having our data channel open.

And now that we have our data channel, we can send our command to blink the LED. This is basically the hello world of hardware is blinking LED. We see it happening over in our camera stream to our right. This is great. It's glad that I can see it in the same room as me. But what if I wanted to see it somewhere else in the world?

Well, let's use what WebRTC has to offer, which is video streaming. I have a webcam connected to our Raspberry Pi over USB. And so I can start sending that video stream from the Raspberry Pi to my browser. I can start my video. We can see now the exact same thing happening to our overhead cam happening straight from the USB webcam connected to the Raspberry Pi.

This is great. We've got the basics done. But what about something a little bit more realistic? Well, in New York City, the VM office has a claw game. And this claw game is an enclosure that has a camera up attached to the ceiling of it, as well as a USB webcam attached to the claw itself. And this robotic arm can be used to move around and then pick up balls and drop them back into the place like any other claw game.

And we have a TypeScript web app that can be used to control it. So I can go and pick a coordinate within our claw machine. And we can see happening from 20 miles away that the claw starts to move after receiving this request. The IP camera is a little bit slower to update just because of frame rates. And then I can request that it goes and picks a ball and moves back to location.

This is all extracted by TypeScript APIs. We can see it picking up a ball. We got a little lucky this time. Just because it's a robotic arm doesn't mean that the claw game will always succeed. And bring it back to our bin. And eventually drop it in place. Now, you may not be surprised to learn this is exactly how VM enables developers to build robots and other sort of smart machines, along with a bunch of other features.

The robot development kit is the open source core of this product. And it's supported by the cloud services for configuring, controlling, and monitoring your single machines or multiple machines out in the field. The type two SDK, like I demonstrated with that web app, is used to control active machines, camera streams or even querying sensor data from the browser or node JS environments. When you're done checking out the code, you can try VM in the browser without needing hardware in hand.

You can rent a rover for free for a short period of time. You get experience of controlling, programming or even training a robot from the comfort of your computer. Hopefully, you're sufficiently motivated to build a robot for yourself. So VM docs are a great way to be inspired and learn how to make your idea a reality. Or explore our code labs for end-to-end tutorials of all kinds of concepts, from hardware automation to computer vision to speech control. You can explore the demo code from this talk on GitHub. Thank you for listening. And you can see what the VM developed community is building by joining our Discord. Scan the QR code to stay up to date with what we're working on at VM, along with the latest industry news. You can connect with me on LinkedIn at Nick Hur or at Hips Around Most Anywhere Else. Happy building.