React in the Autonomous Robotics Industry

Hello everyone. I'm Hamza Hawi, software engineer at the Autonomous Robotics Research Center at TII. And what led me to do this talk today is that I got too many questions about what JavaScript developer is doing on the Autonomous Robotics Research Center, or how can we combine JavaScript and robotics, or what's the relationship between JavaScript and robotics. And in order to answer this question, I will start by defining who we are.

So I'm part of the Autonomous Robotics Research Center at TII, and what we are trying to do is to research, develop, and innovate new technologies for robots in order to make them fully autonomous or perform some specific tasks individually or all together combined. And we have a mixed fleet of robots, which can be autonomous cars, drones, or even boats. And we have also racing cars. And in order to put some context to the discussion, I will show like an architectural overview. So we have a mixed fleet of robots that are sending data continuously to the server, and that data will be sent to the client side via web sockets. And for the client side, we may have control stations, laptops, tablets, or wherever is needed to take control over that robot.

So now React comes into play because we are trying to develop what's called a mission planning software. A mission planning software is a software that lets you, let's say, optimize a road or draw a specific path or goal for the robot to go to autonomously. And if we try to see the generic screenshot that I have on the left side, we can see that we have three core parts of the system. First will be the maps and the geographical information. And in order to have clear view about what's happening, we need to have some streaming coming from the robot. That's why we need multimedia components. And if we want to, let's say, drive that robots and the operation mode, we will still need to read some data from the joysticks that we have. And that's why we developed some control components to take control of that robot.

Now why we decided to choose React. As we know, robotics applications need good performance and responsiveness that will let you operate with, let's say, operate easily those robots. And if we try to see and analyze our software, we will see that we are receiving at minimum 10 messages each second for a robot from data messages or health of the sensors. And we also have multiple vehicles operating on the same time. So at this point, performance is very crucial. And that's why we decided to use React because React, due to its memoization and the component lifecycle methods, will help us to avoid unnecessary renders by choosing exactly when each component should re-render. And also, because of the memoization, the memoization will give us, let's say, the ability to reduce some computational power just by caching the operations that we did before. And one of the key things also is that the huge community support that React has, because as you know, React is here for many years, and the community is growing and you will have a large choice between libraries and packages that you can use. And this will help us to avoid to re-invent the wheel at each time.

Now let's start with the first part, which are the maps. As you know, when you want to choose a map to start working with it on React, we have multiple choices. But for us, we decided to choose leaflet.

Leaflet, as you know, is an open source package and it has multiple plugins and it gives you the flexibility that you need to operate on multiple types of vehicles, drones, cars or boats or whatever. So that flexibility helped us to cover maximum use cases that we can.

Now let's start by the first functionality that we have. As you know, we need to send goal or path to vehicle and if we are operating under drones, we need to define the zone that we are operating to. For that, we use the package called react-leaflet-draw. So it's a package that came with a long list of, let's say, drawing tools from, I don't know, from lines, circles, rectangles, markers, that you can use. And it came with multiple set of callbacks for each event, for the creation or even when you delete or edit one of the paths that you draw before. And it will give you the vector that contains all the waypoints drawn before and you can use them whatever you want based on your use case and your application.

Now, what if we want to reduce the amount of internet that we are using or what if we are operating without internet? For that, we decided to use a package called react-leaflet-offline, but before talking about that package, let's just try to see and know how leaflet is rendering the tiles, which is the layout that we put on the top of the map. And if we see this, we will find that React is not rendering the whole map on the same, is not bringing the full layout on the same time, but it's bringing them as a set of rectangular images and loading them. And this will let us avoid to render a full map at the same time. So if we want to cache those images, one possible approach is to create your own server that will send those images based on X, Y, and Z, which is the zoom level, and send them to your frontend. And another approach also is by using what's a package called react-leaflet-offline. As you see here on this example, when using react-leaflet-offline, you will have the choice to save a specific area that you are zooming in right now or delete that area and save it again. The map is updated, if there is any update on the map, and on the blue, those rectangles on the blue, as we see here, those are the areas that we save at different zoom levels.

And for the next thing that made us choose Leaflet is that on some use cases, maybe you are not operating on Earth. Maybe you are creating some aerospace application where you are operating on different map than the map that we are using to. So React-Leaflet gives us the ability to create our own, let's say, coordinate reference system based on the requirements that we have. So for this, you can also define how many pixels you will have by each meter and put your origin of the map. You can choose the origin, define it as you want. Even the coordinate system, you can put your own coordinate system based on your use case. The second thing is that after defining exactly your coordinate and the reference system, we can put on top of it our custom layer. For example, if we are operating on a warehouse or a room or any indoor area, we will have to provide the map with our own tile. For that, you can put what's called on Leaflet, image overlay.

Now if you are wondering what we can do with a video overlay, for example, if we are on some weather forecast things and we want to show the direction of the wind. So for this, you can put in a video overlay that you will specify exactly the bounds, like the top right and the bottom left of the video. And it can be as we see on the right side of the slide. This is like a video overlay of a weather forecast where it will be moving and you can see the wind's direction and all those things.

And it can be as we see on the right side of the slide. This is like a video overlay of a weather forecast where it will be moving and you can see the wind's direction and all those things.

The next thing is that, as I mentioned on the second slide, we have a mixed fleet of robots, which are boats, vehicles, and even drones. And this means that we will have different types of joysticks because we can't, for example, still operate a drone with a steering wheel. So we will need to find a way to optimize this.

And for that, we use strategy design pattern, which was a very good option for us. As each, let's say, each robot, we receive the same, let's say the same events, but the axis that we are reading data from the browser gamepad API is different. And that's why we decided to create what's called the joystick strategy. Then one thing that you can do is that to create a joystick context that will render anything just to read the data by setting an interval that updates each specified number of milliseconds and send that data continuously to the robot or the vehicle.

So that's it. Thank you for your attention. And if you have any questions, I'm still available online on all platforms as mentioned. And thank you.