T3Boy: Making Accurate Game Boy Emulation Accessible with Next.js and WASM

Hi, this is Travis TI Kevin83 McGeehan. I'm a senior software engineer at Gordon Food Service, and I'm an ambassador for Task Videos and administrator at Taskbot. I'll explain a little bit about what those are in a second, but this is a talk on TypeScript in speed running and the T3 boy emulation front end. So I'm going to go over who the heck am I, what Pokemon is, just in case nobody knows yet what speed running is, and then we'll talk about T3 boy, the new emulation front end I built in the React and Next.js stack with WebAssembly.

So who am I? I'm a specialist in React Native and GitLab CI CD at Gordon Food Service, showcasing React patterns at JS World and React Summit. I am a task record holder in Pokemon Yellow Glitchless and No Save Corruption categories, which are just kind of some different categories for longer or shorter runs with different glitches. And I'm a former record holder in the Red Blue Glitchless tasks as well. Others hold records for the Save Corruption category also, where you can win in just one minute by turning the console off in the middle of a save. I'm an official ambassador for Task Videos where we host the records and archive videos of those records for different games.

Now we talked about speedrunning and TASing, but then there's another layer that gets into what I do in these showcases called TAS verification. The idea of TAS verification is to take these controller interfaces and inject inputs into a console programmatically by designing a custom interface for each console. We convert these input logs we built in an emulator to a format the console can understand electrically with how it interfaces to controllers, and we play back that piano roll then instead of in the emulator on the real console.

And for comparison, Mickey Mouse has only totaled $61 billion in merchandise sales, so it's really a really massive franchise. And Pokemon games are RPGs that involve catching, training, and dueling a fictional universe of wild animals known as Pokemon. There's a popular long-running Pokemon anime TV show with some spin-off movies, and Pokemon Yellow, the game that I specialize in and that got me interested in this concept of TAS, is part of the first generation of Pokemon RPG games that were released on the Game Boy in 1996 in Japan and 1998 in the US. The first generation had more bugs and was less well-connected to the rest of the overall anime universe than they eventually changed in for Yellow. And there are now nine generations based on different regions of Japan, and recently also some different countries.

So I've talked a lot about speedrunning. What is speedrunning? Speedrunning is the idea of playing a video game often, but not necessarily a retro video game, with a timer running to track the completion time. Speedrunners add arbitrary completion goals to games to enable timing, creating a variety of potential categories of different lengths, i.e. beating a game with or without glitches, or if you want to beat in the Pokemon, for example, if you want to beat the Elite Four, or catch all the Pokemon. So you have all these different categories with different timings. And sometimes these runs require external tools that we can write in something like TypeScript, and those help evaluate the quality of the run, checking things like the quality of your Pokemon, their stats, or helping with different categories like tracking how many Pokemon you've caught. So that's speedrunning, but then what is TASing and TAS videos?

Tool-assisted speedruns is the idea of playing through games, not by hand, but frame by frame in an emulator where we can pause and evaluate perfect decision making, ensuring perfect movement and execution of strategies, and rewinding and re-recording these inputs to correct mistakes. This effectively acts like a player piano does for piano music, but for a video game, and these TAS records are tracked separately on TASvideos.org instead of speedrun.com to keep human and TAS runs independent and fair. TASbot is our mascot for doing this process, and this proves that an emulator is accurate enough to perfectly simulate the behavior of a console for all practical purposes. This also acts as a kind of test suite or test Rome, which can be used to do a unit or integration testing of an emulator. Once a input log has been proven on a console, it can now be used back on any other emulator to prove it is that level of accurate.

The idea of TAS verification is to take these controller interfaces and inject inputs into a console programmatically by designing a custom interface for each console. We convert these input logs we built in an emulator to a format the console can understand electrically with how it interfaces to controllers, and we play back that piano roll then instead of in the emulator on the real console. TASbot is our mascot for doing this process, and this proves that an emulator is accurate enough to perfectly simulate the behavior of a console for all practical purposes. This also acts as a kind of test suite or test Rome, which can be used to do a unit or integration testing of an emulator. Once a input log has been proven on a console, it can now be used back on any other emulator to prove it is that level of accurate.

So, talking about all that, what then is T3 Boy? So let's get into some of the user stories about why I wanted to build the site T3 Boy. First off, accessibility. Some aspiring speedrunners might only have access to limited devices. They might be a young person who only has a mobile phone or only a Chromebook that they can't install apps on. They don't have admin privileges to the Chromebook. So they can access websites, but they can't install things. Also, there are already mobile and browser emulators, but they don't have the level of accuracy needed to do playback of the kinds of things like TASes to the level of accuracy that we need for our verification. They might not support the official BIOS being uploaded by the user. We make sure that the user uploads that for copyright reasons. But if they do upload it, then we can have that additional accuracy that we need for something like TAS verification. And there's just other details of emulation accuracy that are not commonly supported in browser emulators.

One other thing that I love to talk about in this context is speedrun preservation and TAS preservation. So normally on TAS videos, when we have a record finalized, we create a video of that TAS record to put on YouTube and to archive on the website. But this is a costly, expensive, and also a lower quality because video has to be compressed to be sent to end users. So one of the things I thought of with building a web emulator is that the sum total of the game's Rome, that BIOS I talked about, and the emulator can act like a video player when given an input log instead of a video. So this allows us to kind of recreate the video at the end user's computer and upscale it directly for their screen, increasing the video quality of the playback. And another way you can think about this is it's something called Kolmogorov complexity in math. Reducing the sum total complexity of the data needed to recreate a program is the idea of Kolmogorov complexity. And sometimes you'll see challenges for this on the Stack Exchange CodeGolf website where a specific challenge has the idea of reducing the overall complexity of a program. So other problems about doing this. So imagine we wanted then to make a web emulator more accurate. Historically, the emulators that are accurate enough for speed running have been limited to desktop PC builds and the equipment necessary to stream runs done on the original hardware instead of an emulator is expensive. You got to buy all the specialized equipment to be able to capture old video game hardware. So emulator cores have typically been written in low level languages like C and C++ to be able to run fast enough at the original speed of the console. And while C++ itself is pretty portable, the UI front ends we built around it are not necessarily as portable.

We've done things like Qt in the past to design these emulator front ends. And Qt has only recently started having web assembly accessibility. So that's getting gets into the benefit of incoming web assembly. So WebAssembly is a sandbox execution environment for binary code. Its main use is for websites to enable running high performance code directly in the browser by having a little virtual machine for high performance code. And scripting is the technology that allows us to take C or C++ code and compile it to WebAssembly, meaning that now we can take our high performance emulation and highly accurate emulation cores we've built for the desktop and package them to be run in a website. And the front end UI can then be written in JavaScript or other common web frameworks like React or Angular and interface with the web wasm core over hooks in Emscripten.

And that's what we're calling T3 Boy. So T3Boy.Versel.app is a website you can follow the QR code here. And the idea of this is to do exactly that, package the existing Gambat, Speedrunner, now GSR core into a web front end instead of a desktop front end. And by integrating with the T3 stack, TypeScript, Tailwind, and TRPC, we're able to use things like Discord OAuth to back up your game saves and save states into the cloud via authentication to your user. And the emulator also has some really cool custom pointer events API usage that I'll talk about later to do some fancier things that wouldn't normally be possible in a basic JavaScript web emulator. And the emulator also sizes itself and its video output according to the same principles I previously demoed for RGBscaler.com to maximize the video quality of the output like I was talking about for preservation purposes.

So this is what T3 Boy looks like. You have your button pads, your start select A, B, your directions, and a power button. And I also had to add a play pause button because you can't just start up the emulator automatically when the site launches because of protections browsers have now about automatic playback of audio and video. So I do need a start button. But once that start button has been pressed by the user, they can just play this like they would have a normal Game Boy set of buttons. And just also beware, we're going to be seeing a lot of bad ideas upcoming here at that work. One thing I've learned in demonstrating things about web technology is that if you want to learn something the right way and the internet, the first thing you have to do is show how to do it the wrong way.

So we're going to be showing you some wrong ways of doing things. And hopefully we can all learn together from people giving feedback in the future after this talk.

The first thing I did was I integrated the WASM module into a Next.js boilerplate website. In the head of the document function, I added a script with the source pointing to the WASM bundle, I had it in script and do a single file export mode. And I use the before interactive strategy for loading the JavaScript file. There is supposed to be a way to hook in and lazy load that WASM function, but it wasn't working the right way I was using it. And so I had to resort to just using a set timeout and wait five or 10 seconds for everything to load up before I tried to call into the WASM function.

Getting into the core of how we then integrate with the WebAssembly module, I've declared a global function in TypeScript, which has the name module. So it's a global module, but Emscripten named their thing module. So it's called, it's actually called module on the global scope. And it has these functions necessary to interact with Emscripten's bindings for the C code. And I'll walk through what each of those bindings does. So the first really important one is cwrap. Cwrap returns a JavaScript function that can be used to call exported functions in the JavaScript module.

Cwrap returns a JavaScript function that can be used to call exported functions in the JavaScript module. This takes arguments as the function name that you want to call in the C module, the return type of that function, an array of argument types. Annoyingly, this is duplicated with the value of TypeScript generics that can type the return function. It seems really like Emscripten was written before the advent of TypeScript. So there's some duplication there. But cwrap does then give us back a function like, for instance, gamback create. Gamback create gives us the actual full emulator object, which really it actually just gives us a pointer.

A pointer in C and lower level language is just a number. We don't need to think about it any more complex than just, it's a number that is used to reference a certain object in memory. Cwrap gives us back that function which can be called to get a number referencing our location of our Game Boy emulator in memory. Then there's three other concepts important for memory management. In JavaScript, you don't normally need to worry about memory management, but because we're interacting with a C library we need to do some around memory management. So malloc, heapuate, and free are the big ones.

Malloc allocates memory for us and gives us a pointer, again a number that references the start of that memory section. Heapuate is a Uint8 array in JavaScript that accesses and sets blocks of memory at a time instead of individual pointers. Every malloc needs a corresponding free so that you don't leak memory. There's this really cool interaction of concepts between how JavaScript does clean up and how a low-level language does cleanup, where in React you might have a use effect where you do a malloc when something changes and then in the return of the use effect where you do your cleanup you would do that free then to make sure you aren't leaking memory.

Every time it does a new malloc it does a corresponding free. In this example here we need to get a way to load the BIOS because, like I said, the BIOS of the Game Boy is something the user needs to supply because it's copyrighted by Nintendo. We can't just provide it freely on the internet. We say allocate me a section of memory the length of the BIOS, set the data we have for the BIOS into the heap area at that location, and then tell the emulator to load the BIOS giving it the pointer to the emulator object and the pointer to where we loaded that data in. And then after that's all done and said, we free the BIOS data as RAM so it's not being used.

Another example is the setValue and getValue functions. setValue and getValue are like heap U8 but better for working with individual 8, 16, or 32-bit numbers. We use setValue to tell the emulator core how many cycles to try to run for, which is useful because we want to go for about a frame at a time and render that frame audio and video. And we can tell it, OK, go 35,000 cycles. That'll get us a frame. And then we use getValue after the emulator has run that frame to check how many cycles it actually ran for because it can't actually run for an exact number of cycles. It does some event-based behavior that means it might need to overrun that a little bit, so we need to know how many it actually ran for and we get that back after we run the emulator. And yeah, setValue and getValue are useful for getting those kind of individual values instead of trying to access these big values from the RAM. And similar things for getting the video and audio. We could be using heap U8, but the video comes out in the wrong red, green, blue order, so we really need to go back and make a modification on the branch that is a WASM version for the video to come out in that other order so we can use the full heap U8 and just pull out a block of video at a time.

Anyway, moving on, there's also add function. Add function is really useful for when we need to have a callback function. In the C library, you might give one thing a function that is somewhere else in C, but we can also in scripting give it a JavaScript function that then gets ported over to the WebAssembly side. So add function lets us do something like give a button retrieval callback to the emulator that it can call to get the current button state, because the way the emulator works is that you can't just send it buttons whenever you want. When it gets to the point where it needs to read buttons and it's looping, it needs to call that callback function. So we use add function to get the function. We get a button function pointer. We use C wrap like we did earlier to get an input getter, and then we actually assign the pointer to that add function location to the input getter. So a little bit complicated there, but we'll see a bit more why we need that input behavior too in just a second. So another thing I wanted to talk through was some interesting quirks with the Web Audio API I ran into when building this, which is that the Game Boy's audio comes out really weird for most emulators. The Game Boy was a 2 megahertz audio system, and I think it's like 2 megahertz 1-bit audio or something crazy like that. And re-sampling that is really complex because browser's only natively in the Web Audio API, except up to 768 kilohertz, and that's browser-dependent. It's not a spec thing. Every browser has a different limit of how high their Web Audio API can get in terms of re-sampling natively. So in the future, to fix this, we're going to need to change the C library underlying the site to emit pre-re-sampled audio to 48 kilohertz, which is a more normal frequency for an emulator to be using and for playing back in the web.

Another thing, and this is a really great reason why I wanted to build my own emulator site for this, is using the Pointer Events API. So there's this concept called pointer capture, which is like, if you change a volume slider or scrobbling through video, you don't want the movement of your finger to go up or down a little bit and move off of the scrobble and no longer be dragging it. So there's this thing called capture, where once you're moving it, you are still moving that slider no matter where your finger goes on the screen until you release your finger. But in the case of Game Boy inputs, we don't want that pointer capture because we want to emulate the behavior of like, for instance, you put your thumb down on the up button and you move it to the right button. You don't pick your thumb up and put it down on the right button. You might move your thumb directly between the up and right button without it ever being lifted off the screen. So in order to emulate that, every time we get a pointer capture event, we have to immediately release the pointer capture. And that allows us, then, that when we move our thumb from the up button into the right button's box on the screen, it detects that new target and says, hey, you're pressing the right button now to our pointer events. And this is huge because also the Pointer Events API allows us to do something like pressing the A button on the right side of the screen and the up button on the left side of the screen. And then we can do multi-touch control and do all of our advanced manipulation methods that require very precise movements of buttons to get to places on the screen and to manipulate the RNG.

Lastly, another really cool feature of T3Boy was enabling cloud saving. Discord Auth and a quick railway database enabled cloud save backups on the site for authenticated users. And I implemented this for free at the moment. So if you guys go blow up and use this a bunch, I might have to lock it behind a Patreon subscription or something. But for now, you can go out and use T3Boy and have cloud-backed Game Boy saves on the browser. Thank you guys so much for watching. This has been a walk-through of the basics of setting up a WebAssembly integration in a React website with the T3 stack. And I hope you guys enjoyed it. See you next time.