React Beyond the Browser

Hi there, welcome to my talk, React Beyond the Browser. I'm Menahi Shayan. I primarily work on open source projects in the Boston area. I've been formerly a CTO and co-founder of NextUp. I was formerly the founding engineer of ZenDuty, and I will be joining the product management team at T-Mobile.

And today we're going to be talking about React Beyond the Browser. Now we're JavaScript developers, right guys? We love everything JS has to offer. It's really easy to learn. It's developer friendly. It's loosely typed. Well, loosely. It's got rapid prototyping. It's got a really helpful community. The ReactJS framework, TypeScript, NPM. The largest package ecosystem in the world, by the way. We love it so, so much. And what do we want? Not much world domination. The usual. Right?

No, but seriously, think about the fundamentals of the React native ecosystem and what it stands for. Learn once, write anywhere. And that makes sense. So JavaScript, in my opinion, is one of two languages that makes the power of code more accessible. So it would only make sense that being able to use React in any software development scenario that would require a UI would make sense for us. It would make things incredibly easy. It would streamline the experience. And it would help more developers get started on more platforms with minimal effort. So what does it look like today? Are we there yet? Well, no. No, we're not. We are nowhere there yet. Not everything is JS.

And as a JS developer, there will come a point of time in your life where you will have to step out of the JS comfort zone. Especially if you've got to build tools for other apps. Take, for example, plugins and extensions for common apps on the marketplace. So for example, for a game engine, you'd be writing code in C sharp. For OS programs and tools, you'd probably be writing bash. If you're writing plugins for the Adobe suite, you'd probably write with its own extended script. There's a lot of us that use .NET and C++ embedded systems, maybe C, Python, office add-ins use Visual Basic and so on and so forth.

And with all of these diversity in creating different UIs for these niche platforms, there comes new paradigms, new syntax, new restrictions. Let me give you an example of what I mean by taking office add-ins with Visual Basic. For example, this is what a snippet of a JSON parsing module looks like in office. And right off the bat, there's so much going on with massive string parsing, having significant digits, low level memory management, completely new paradigms and syntaxes that most developers are just not familiar with. In fact, most JS developers don't even deal with low level memory management on a day-to-day basis, not to mention the IDE that you're restricted to.

And this is just the CLI. The UI is a whole other thing on top of this. Now imagine learning all of this from scratch for a very specific use case. Think about how much time you'd have to spend, how much context switching you'd have to do just for all of this. Think about the complexity of new paradigms that this brings with it. Well, luckily, we don't have to do that anymore. And that's exactly what I'm here to show you. We're going to try to build office add-ins with, are you ready? React. Yeah, that's right.

We're going to take native's philosophy of learn once, write anywhere to a whole new bounding box. And let me show you how we can do that. We're going to get started with something called Yeoman, and we're going to have a quick generator that we're going to install, and we're going to run the office generator using Yeoman.

Now, this actually takes care of most of the work for us, where we can just sort of select what kind of template we want to get started with. I'm going to go over here and select office add-in task pane using the React framework. And we're off to the races. It's going to set up all of the boilerplate that we need to get started. And let's see what it looks like.

Yeah, that looks pretty good. So we've got like a nice little add-in task pane that is set up for us. It's got it up and running. It's got some text. It's got an input field, and it's got a button right over there. Let's look at what the code base structure for this Excel add-in looks like. Relatively familiar, yes. We've got a task pane folder over here. We've got the components folder, and that's where most of our magic happens.

The app.tsx file, as you can see, we're writing in TSX in Excel. Yeah, that in itself is a noteworthy mention. And we've also got our taskpane.ts, where our non-UI logic is going to go. Now let's go ahead and tinker with it a little bit. Try to familiarize ourselves with how we can take advantage of this. Now, for example, for SARS, I'm just going to change some of what the UI looks like. I'm going to say best conference ever. And there you go.

The HMR has kicked in and updated that title in Excel. You don't have to build your VBA all over again and none of that stuff. Let's take this a step further and see what you can do beyond just React. Because we have an incredibly powerful tool in our toolbox, the ExcelJS API. Yeah, we can directly access the entire Excel sheet and manipulate it as well.

In this example that it's gotten us started with, it's using a simple get active worksheet to find out the sheet that we're currently on. It's trying to access that range and trying to add values, insert specific text into that range in a two-dimensional array format. Finally, we're going to call something called the context.sync, which since all of this is an asynchronous operation happening, and our JS is communicating with our Excel runtime, we need to have a sync function that sort of communicates both ways and executes updates in both directions so that the context is synced between our Excel and JavaScript runtimes.

Let's try writing something on our own. Now we've got our insert text function. Now let's also try to grab text from what's been selected or entered onto the worksheet. How do we do that? Let's take inspiration from this. Now I've got a boilerplate function ready for me right here, and I'm going to do something very similar. So I'm just going to get the selected range from the workbook. I'm going to now extract two parameters from it, values and address.

So I'm just going to get the selected range from the workbook. I'm going to now extract two parameters from it, values and address. Now the reason we need to do this is once you get the range, Excel sort of handles attributes differently. Because there are so many attributes, it doesn't actually query everything all at once. You query attributes on demand. So at the moment, I just want to access what's in the cell, the contents themselves, and I want to access the address that is where they are within the worksheet.

I'm going to call context.sync again, and I'm also passing a helpful setState function where we're going to send that data. Now that data is going to be sent on the JSX or the TSX side of things, and let's see what that looks like. I'm going to create a simple new state, and I'm going to call our function that we just created in our use effect and then call the setState function so that we can sort of set that data into the state. We're also going to render that data in the DOM itself using our p tag, and let's see what that looks like.

Yeah, there you go. So I've selected my F3 to F5 in my Excel sheet, and the moment I do that, I can see that JavaScript or React is able to access the data that I've selected. It tells me the address of what I've selected and the values that are in those exact cells. But this is nice, right? But let me tell you where the real magic happens, where we can truly leverage the full power of writing these add-ins in React or JavaScript.

We can take advantage of 3 million NPM packages. Right off the bat, to build something practical, I would, off the top of my head, use Google's generative AI just because, and see what we can do if we can quickly hack together an AI app for Excel using the power of React. Throw in some generative AI API calls through an NPM package, add another package for Tailwind CSS just to make our code look really pretty. Our UI looked really pretty with minimal CSS code. And also, we could even use Fluent UI's own React components if we want to make our UI more consistent with Microsoft Excel's interface. And combine three of these together, throw in some chemical x and Professor Utonium, and what do we get? There you go.

Yeah, so we've got a nice little Excel add-in on the side. It's got a nice little description for what it can do. We've got our selected range that looks more discreet right now, and it's also in a very nice shade of blue. We've got an input field and a button over there. Let's see if we can hack this together and what this does. So, I'm just going to type in, give me a list of bank transactions, send this to the AI. And there you go. Through the power of JS, I've made an AI API call, entered it into an Excel sheet, and I've also got a little helpful description. I hope that's a helpful description of what the AI did.

And a little tool tip that's using Fluent UI's React component library to make it look incredibly consistent with the rest of Excel. And if you've noticed, it's also in dark mode. It's also aware of the context of the rest of the Excel application. Just to take things a little further, let's see if we can ask an additional prompt, not just to modify text, but to also read text and also then add text right next to that. And since we were aware of all the context within Excel and where things are, JS is able to modify things into the right places. There you go. Learn once, write anywhere. We've got the power of React running within Excel. In just under a few hours, we were able to build a fairly comprehensive application for Excel using React. But let's see how far we can push these bounding boxes because anywhere does mean anywhere. Imagine using React on embedded systems.

Yeah, so this is an LCD display, a 128 by 160 embedded systems LCD display that uses the IO line 9341 controller. And for the uninitiated, it is rather bare bones. So you've got a, it's usually like one to three inches in size. You've got between like 128 to like 320 pixels. So not too many that you're working with. You've got, you do not have an HDMI or VGA port that you can just plug into and like extend your display or mirror your display. It's only controlled through something called the serial programming interface, which is like five wires connected to a microcontroller. It's got barely 150 kilobytes of RAM, usually controlled through an ESP controller like I have over here or an Arduino board or a Raspberry Pi.

And all of the rendering is usually done using C++ or Python. Now to give you an example of what the traditional way of rendering looks like on this, this is what you would write in C++ to generate UI on this display. So it would be a lot of like draw a rectangle here, draw a circle here and exact pixels of like, this is our X coordinate, that's our Y coordinate, this is our height and width. So it's a very specific drawing instructions, kind of like how we do on the canvas. And this is sort of what that would look like. Can have some degree of text and very specific drawing instructions. And similarly in Python, this is what we would do, the equivalent of that. But as you can see, the syntax is largely the same. Specific canvas like instructions on where to render, what to render.

What's missing here? For starters, where's the CSS? What about state management? How do I trigger updates and refreshes? What about animations? I want my content to look smooth. Where's the JS driven DOM updates? Design library, tailwind, DOM, gifs, anything? No, what's the dream? The gold standard, if you will, React, right? Or other flavors of React. Now, do you think this dream is achievable? Would we be able to achieve React running on a display like this, where a lot of it is on just a few pixels, no HDMI or VGA boards, SPI interface, just a few kilobytes of graphics RAM. And all of your rendering is being accessed through C++ or Python, which means you don't readily have a browser available on this. And the biggest of all, you're entering one frame at a time. How do we solve this problem?

How do we solve this problem? There are quite a few solutions out there. Some of the most popular ones are still contained within Python, where you've got a browser running. Let's say you've got your React app running on your local 3000. And what Python is doing is screenshotting every single frame and resizing it and then sending it to the display. Now that can be incredibly inefficient and use up a lot of resources, but you're also not going to get smooth animations because this logic is going to be bottlenecked to about three frames per second. Other approaches to this problem online tend to include starting up your browser or having your React code running in it, extracting the current state of the DOM, traversing through every single attribute within the DOM and recreating those attributes in an equivalent Python package that you can then send to the display.

But think about that for a moment, you're essentially writing your own browser. That is intense. You have to account for every single possible state and attribute that DOM elements can have. You're not going to have a good time doing this. But I'm about to show you something today, a radically new solution. And this is going to be a complete game changer that's going to allow you to render React on embedded system displays. And this solution involves directly working with the Linux system frame buffer, where we're going to set up that as a secondary display and have Chromium run on it. That means we're going to have HTML, CSS and JavaScript out of the box with smooth animations, great framework support, and it's going to also support like partial display rendering or updating only specific parts of the display since it's a system control frame buffer.

And we're going to get high FPS. How high you ask? In testing 52 frames per second. Yeah, that's smooth. How do we build this radically new solution? Well, I'm not going to go through too much of the specifics of this code because it sort of digresses from the scope of this conference. But you would have to first set up the driver in Linux, you'd have to install X11 and Chromium. You'd have to also set up a frame buffer for the display in X11, have it configured to start up on boot, and you'd also have to configure your X11 to sort of run Chromium along with all of your node modules if necessary. But the end result... Ta-da! Vercel offered me a cookie on an embedded system display. How cool is that? Right? Now, in this example right here, I'm actually running a React-based package called Magic Mirror on this embedded system and I'm able to make seamless API calls through JavaScript.

How cool is that? Right? Now, in this example right here, I'm actually running a React-based package called Magic Mirror on this embedded system and I'm able to make seamless API calls through JavaScript. I'm getting my full HTML, CSS, and JS UI rendering here. In fact, I can even open the Chromium inspector and I have full DevTools access here. I've got all of my CSS loaded up, I've got... I can inspect my DOM, I've got animations, I've got tailwind. In fact, I was even able to play video on this and it was fairly smooth. I can also take this to the next level and the icing on the cake over here is having Chromium DevTools remote... with a fully remote debugging experience on a laptop while having HMR. This may seem pretty common on a laptop but this is unlike anything we've ever seen in the embedded systems community.

But you know what the best part is? We can leverage every single package that you can find now beyond your browser. We can unlock the full potential of JavaScript on an embedded system. I want you to go ahead and try this. So this is incredibly rushed set up tutorial but there will be a QR code after this that you can scan and access the GitHub repo. But I want you to try to set up React in an unconventional place and see what we can achieve by unlocking these limits beyond the browser and truly take the native philosophy of learn once, write anywhere to a whole new bounding box so I don't just want to know what you're going to build. I want you to know where will you build.

So if you're interested, go ahead and scan the QR code check out the repo to run React beyond the browser. That's been it from me. I've been Minay Shahyan. Thank you for listening to React Beyond the Browser.