Rise of the Robots

Hi, everyone. I hope you're enjoying the conference so far. Okay. So I'm Theodor. I'm a software engineer based in Athens, Greece. I'm also the founder of Proxima Analytics. It's an open source ethical first analytics platform that you should definitely check this out. And you can also find me online via the alliance wordless. So if there's a question that we can make for 2023, it's going to be are there robots going to actually take over? Like, is going your freeds going to take you as a hostage, or are you going to lose our jobs as software engineers? And truth to be told, this question is, like, rather too old, if we can say so. So this is Isaac Asimov, one of the most famous science fiction authors. And in one of his books, which is called iRobot, in 1953, I think, he came up with the three laws of robotics. So basically, this is a manual when robots are going to actually take over. And today, we're going to challenge this question, and we're going to try to find out if this is true or not.

So fast forward, in 2018, I was working for a company. And besides that, I was also trying to experiment with different aspects of software engineering, trying to mix up things like 3D printing, electronics, and stuff.

And I was working for a company where we had three major products. Like, the first one was a web application and two mobile applications, one for Android and another one for iOS. And I can really tell you this. So mobile end-to-end testing is pretty damn hard. It really is.

So if you have ever tried to run end-to-end testing in mobile applications, we're basically more or less stuck to the emulator. On the other hand, we also need to test mobile applications on real devices, right? But it's quite tricky, actually. So for security reasons, most of the applications are sandboxed. That means that we cannot actually test how our applications are interacting with the operating system. We cannot test sharing links between applications. We cannot test a workflow where we want to authenticate users using the email client in the mobile phone.

Moreover, we have external interruptions like phone calls. Mobile phones are like living organisms. So we have phone calls, notifications, push notifications, and so on. And then I had that weird idea, like, what if we could actually try and build a robot for automating tests in real devices? And you know what? I know that most people would actually think something scary or big like the Terminator or so. But truth be told, we can categorize automating automation robots in three big categories based on their movement.

So the first category is they're called Cartesian robots. We have three axes. And basically, the actuator moves into the three-dimensional space using bolts and wheels. 3D printers and CNC machines are working in that way, in that format, actually. But in our case, it does not work because you have a limited area to work with. And also, the movement does not feel that natural.

Next up, we have robot guns, which are basically made with separate motors and three or four attached parts. They are a needless standard for the car industry and medical operations as well. But they're pretty hard to operate. And moreover, they're pretty expensive and mostly used for repetitive tasks. So our pickup for today are called Delta robots. We have a base on the top, three or four motors attached.

And as you can see, we have pairs of arms that are connected to the actuator. They're pretty easy to operate. Basically, we can use basic trigonometry in order to calculate the position and move the robot along. And the movement is fast, efficient, and really natural.

But how we can actually build that kind of a device, right? Luckily for us, there is a company called Tapster. And it has been founded by Jason Huggins, who is also one of the creators of Appium and Selenium. And two of their devices are actually open source. So this is called Tapster Bot. This is the version one. And based on his design, I actually forked the entire setup and the code. And today, I'm going to present you a real device. So this is Margaret, as you can see here. Thanks so much. By the way, this whole setup runs inside my browser, so this is a live feed here. As you can see, we have arms attached here. Most of this part is using bolts, nuts, and screws, and 3D printed parts as well.

The brain and hearts of this device is a tiny microcontroller. But you might be asking, where the hell is JavaScript? This is a JavaScript conference, right? So truth to be told, JavaScript is a really good option if you want to program stuff in microcontrollers. There are a bunch of frameworks out there like Device Script, Joni 5, Notebots, Esprino, you name it. And JavaScript is a really good language because we're handling events naturally, callbacks as well. So this is the things that you're doing with a device like this one. In our case, we're going to use Joni 5.

The brain and hearts of the robot is actually an Arduino nano microcontroller. So Joni 5 works as a proxy. The device is connected to my workstation, and then we can directly run a Node.js server in the backend, and we can send commands that the robot executes them. So this is a basic class that allows us to run a really basic robot. We first have to instantiate the Joni 5 board. The microcontroller has labeled pins. So we can say that we're running three server motors on pins two, three, and four. And then when the board gets instantiated, we can run the actual class.

And here is the very basic interface implementation of doing so. You have the initialization function, and then some helper functions like tapping, moving, swiping, and resetting the device. And I told you that this is live, right? So since we have a Node.js server in the backend, we can do whatever we want to.

So here I have... We can expose REST APIs or even web sockets. So here, as you can see, I can directly move the robot through the x, y, and z-axis as well. Or I can actually make it dance for a while. The basic movement is more or less sending commands and coordinates, like go to point 0, 0, 0, and so on. Let me just unfocus this one.

So if you think about it, we can move the robot just sending the coordinates within an interval. If we're talking about taps, like tapping the actual device, that's just about lowering the z-axis to 0, like touching the actual device on the bottom of the robot. And here's the basic implementation of doing so. But there is one thing here. Because that device does not exactly know where the mobile phone is, we need to actually calibrate and let the robot be aware of where the mobile phone is. In order to do so, I have created a really simple web interface. So we can start the calibration process. The robot lowers down to the device, and when the touchscreen gets touched, it sends a command back so we can take pinpoints and find out where everything is right now.

Now let's talk about swipes. Now that the robot is actually calibrated, swiping is just touching the screen to point A, dragging to point B, and releasing. In order to do so, I have recreated the Tinder UI, you know, the one with the cards. So we can start picking our mate for today. Cool. Let me unfocus this one. Okay. So now that's everything we actually need to do. So we have taps, and we have swipes. Since we have the Node.js server in the back end, we can do whatever we want to. We can direct with the REST API. We can use a web socket. We can actually also use Opium.

In this example, we're just instantiating the Opium JS SDK, and then send directly commands back to the web server. So everything is in place, and we can start testing out our application.

But I know that you want the full-fledged demo, right? But as I told you, today we're gonna challenge the real robot. So I want everyone in this room or remotely to pick up your phones. That's a new one for a conference, right? So you're gonna compete against the robot.

We have a really simple game here. We also have two big prizes, right? If the robot wins, we're doomed. Try to find out the bunker. Try to run away. But if someone of you wins, then there's a full-year sponsorship by Egghead, like one full year, as well as a copy of Node.js design patterns by Luciano Mamino. I provided this one for one lucky winner. And here's the game.

You're gonna commit with math against the robot live in this room. So you can go to dab.sh slash play. You have to go inside, just pick a username to do so. Please, please, people, be kind. And once everyone gets logged in, we can start playing around. The game is really simple. You have an equation, like a mathematical addition. You have to pick up the right answer as fast as you can. Time matters here. Okay. I'm just gonna give you, like, ten seconds or so. The whole game will last about one minute. Everyone set? Everyone set? Are we good? Okay. So, are you ready? Just one minute. And you can actually start playing. Okay. It's hard, right? Okay. Almost there.

Five. Four. And we're done. Hands on the phones. Everyone. Okay. Let's see how that went. Are you ready for the results? Yeah. Let's do that. Chatterbox Coder, is someone else? No? Okay.

In the application, we can find your ID that you can send me over through a DM. Congratulations to Rahul, Mikhail, and the robot, actually. But really good job. Actually, Chatterbox nailed it.

Okay. So. This is a wrap. We're going to try and do what else we can do with a setup like this one. So, since we have a full-fledged automation system, we can test deep linking, sharing, like authentication flows, as I told you before. We can run the device 24-7 on different devices, iPads based on the actual base that we can So, we can scale up and down the robot as we want to. We can test post notifications and interactions with the operating system.

But there is also more. So, for example, we can attach a camera and have a live feed in order to detect what the robot touches or swipes. We can get really meaningful metrics, like how much time does our application take to load. This is a really interesting one, because we can take metrics about user behavior or analytics. And because the coordinates and the movement is based on matrices of numbers through time, we can effectively replicate bug scenarios or try to go through workflows. We can also stress test the application, like fiercely tapping the screen and so on. And finally, since the movements can get described really well, we can train our own AI models and we can auto-generate user paths and workflows in order to provide automation testing as well.

So, that's all from me now. I would like to thank you so much for your time and patience.

And I hope you enjoyed the conference so far. Let's give him a round of applause. Let's see the questions.

Before we do the questions, I have a confession to make. I feel really bad about this as well. I am Chatterbox Kodar, and I realized halfway through that I was like, oh, wait, if you just quickly go through questions and you get them wrong, it still speeds up the next question. So, I was just doing maths on the first two digits and carrying one and seeing if it worked. So, I think Rahul should get the prize. So, wherever Rahul is, wave your hands. Give him a round of applause wherever they are, and if they're at home, give Theodore a DM.

All right. That's pretty fair for you, right? Yeah. Okay, so let's talk about some of these questions. So, we'll save the first question until last, but the next question which asks about Axis. If it's a larger device with responsive... I will start that again. If it's a larger device with responsive design, will you need to update the test? Fairly not, because since we're... So for my example, I'm interacting with a web interface, so you can actually place locators and you can identify the coordinates, where's everything. Also, when the device boots up, we get information about the viewport, about the actual device screen, so it's pretty easy to accommodate different designs as well. Tabletop Robotics have also created a base that can actually rotate the device also and tilt it in order to identify, to test the gyroscope and things like that. So no.

Ah, well, thank you. I think that's so interesting how you start with a really simple thing, you're like, oh, can it do this? And you add on the features. It's just like a real product you're developing. That's really cool. All right. I'm going to scroll down to the next question. We will go through some of the cheeky ones later, don't worry, I'll save them for later. But how do you test push notifications? Can we make the robot know that a push notification has been received, etc? Sure. So, in this example, I'm just interacting with a web interface, like placing locators and so on.

TabSir has actually provided a way that you can attach a camera in order to identify what's on the screen and also use a toggle to screencast the results. This allows for image comparison and the ability to detect notifications or receive callbacks when push notifications appear. It's a unique and impressive use of the camera.

Another question is about the speed of the robot and how it affects the duration of CI pipelines compared to a software-only solution. The speed of the robot depends on the movement and precise calibration. It can move quite fast for concurrent taps in different places, as demonstrated by TabSir. They even have a demo of playing Tappy Bird, where the robot taps the screen quickly. It's impressive that the robot can capture images and move faster than most humans.

For JavaScript developers interested in IoT, it depends on the requirements. C++ is suitable for full-fledged IoT applications, but JavaScript is also a good option when attaching sensors or buttons. Jony5 works as a proxy and makes it easy to use JavaScript with microcontrollers. Esprino and Device Script are options for standalone applications. Ultimately, it depends on the individual's preferences and goals.

One advantage of using JavaScript for IoT is that special software is not required. There are different IDs that can be run inside a browser, providing live updates instead of a compilation step. This makes experimenting with Arduino clones and LEDs straightforward and accessible.

Someone asked do you recommend using JavaScript with IoT, as an IoT device has less memory and most devs prefer to use C++ programs. First, let's talk about what you prefer, and then let's give some advice for anyone who wants to get into it with JavaScript. Okay, so it actually really depends. If you want to bring up full-fledged IoT application, yeah, sure, C++ makes totally sense. On the other hand, JavaScript is actually pretty good, because basically, if we have a microcontroller, it's bare bones. It's just a tiny computational unit. But if you start attaching sensors, or buttons and so on, JavaScript is a really good option. Jony5 works as a proxy, so it's pretty easy to use as a workstation. If you want to bring up standalone applications, Esprit and Device Script are really into that. So whatever works for you, that's the proper answer. But if you want to experiment, you can spend a couple of bucks, get a simple Arduino clone, some LEDs, and it's pretty straightforward to do so. The other interesting part is that you don't need special software to do so. So, for example, there are different IDs that you can run inside your browser, and it's pretty cool, because you have live updates inside, instead of having a compilation step and so on. That makes sense. Thank you, thank you.

We've got a bunch of different questions that all really are the same thing, which is about multi-touch. So, zoom, pinching, and different things. What's the future product roadmap, I guess? That's the actual question.

Right now, in order to touch the screen, we have a stylus. So basically, it's a phone pen, the cheap one, like $1 or so, that's grounded to the microcontroller. Basically, you can attach whatever you want to. If you attach a pen, that's a plotter. If you attach a hotend, that's a 3D printer. So, basically, you can attach a small... Like two fingers, like two styluses in order to create swipes and so on. It's based on your imagination and what you are capable of doing so.

Nice, nice. We've got another question, and a lot of these are people seeing that this is great, they want to use it, and maybe there's some extra features that they want. I think the next one's kind of in that, especially you talked about the camera sensor. And I guess if you build a camera sensor, this person's asking about making sure that the next screen that happens after an action is what is expected. And I know you spoke about the fact that there's a camera, the fact that you can use AI. How would you go about implementing that? So, basically, it's based on what you're trying to achieve. If you're using mobile phone applications and you're running the application, you can't just send events in order to make your assertions. Here is a screen. I have swiped. Here is another screen. The mobile application just raised an event, so I can assert that everything works as expected. You can actually also use image comparison to do so. You can use locators. It's up to you how you want to handle this one. For example, for the game that you have actually played, this is a web interface. So every time that I had to make an action, I was just directly sending commands back to the workstation using WebSocket. Nice, nice, nice. And I love the fact that this is because all of the technologies, our JavaScript and things we're familiar with, all the other APIs and all the other pieces of knowledge that we have is available to us to use with it. By the way, I just want to say that the entire cost for this setup is more or less 40 to 50 bucks based on how, if you're capable of, like, if you have a 3D printer, it's probably less.

But all the components are widely available. Like the servo motors are pretty plain, old-school motors for remote-controlled cars. So yeah, it's pretty affordable if you want to do so, if you want to build something like that.

Oh, cool. Because you kind of preempted me on the next question. The next question was what's the price. Yeah, it's more or less 40 or 50 bucks. The cool thing is that right now we have more ways of doing so. For example, Tapster, they were working on a setup where they have actually used a customized toggle that can control the phone using the accessibility settings. So they're tricking the phone, thinking that this is a keyboard and a mouse, so you can even run a similar setup with an Arduino or a Raspberry Pi for way less, I think more or less five bucks or so. So yeah, definitely check them out.

Now that is really, really awesome. And then last one of the serious questions before we get to the fun ones that I know everybody is waiting for. So this one's about if the application UI changes every time you're going to need to update your test cases and do it all quite manually, I'm guessing at this point in time. I mean, is there a future you envision where it could be easier to kind of automate changes to the tests? I think that relies on the setup you actually have. So as I told before about the responsive design, if you... So for example, if you're running tests on the web interface, like using Cypress or Playwriter, so you can actually say, hey, go and click to this position of the screen. That's pretty static. But if you somehow have a dynamic way of identifying the interactions that you want to make, that makes pretty much... Makes the whole setup more flexible. Awesome. Awesome. And one thing I think actually is, and kind of like this answer, everything's an it depends. And honestly, what I would say is afterwards, come find him and have a chat. I'm guessing you'd love to talk about it, considering you've come on stage and chatted about it with all of us. So thank you. Thank you so much. All right. Now what we're going to do is we are going to jump into a rapid fire of yes or no answers for the next question. Is that okay? Yeah, that's fair.

That's fair. Okay. That's pretty exciting. All right. So we'll start with the first one. Yes or no. Will Skynet kill us? In 10 years or so? Yes or no? Yes. In 10 years. Cool. Nice.

Next one. Can it solve the I am not a robot capture? Actually, yes. We have tried this one and it's pretty amazing because they do have the bug there. So for mobile devices it's pretty easy to do so. Fair enough. It's ironic, right?

All right. Next one. Yes or no. No explanations. Have you used the robot for Tinder as well? No. No. Okay. Well, I guess the answer is obvious on the next one. Can you share the source code to use it in Tinder? Asking for a friend. Wink. Emphasis on the wink. The source code is actually open source. It's on the Tapster GitHub repo for Tapster one and two. It includes all everything that you need to build like a bill of materials, a code and so on. And also the 3D designs that you can go to a makerspace and 3D print the entire setup.

So whoever you are, don't worry, you've got it covered. You just need to do a little bit of work yourself.

All right. And next one. If the robot would have won, would you have programmed it to read a Node.js book? Once again? If the robot would have won, would you program it to read the book? I cannot answer this one. Like yeah. Hopefully it just never wins and you don't have to face the problem. Right.

Okay. So this is about the actual, the board, the machine. Is it connected over Wi-Fi? How is it connected? Yeah. So basically I'm connected through, I'm on the Wi-Fi network and both of them are connected to the same Wi-Fi so I can expose like my local setup to the robot, to the mobile phone that's on. That's awesome.

And the last question. It's not a question, but it's one thing that I think we all agree with. This presentation was amazing. You should keep it up. We want to see you again. Give him a massive round of applause. Thank you so much. Thank you.